Commit Graph

2935 Commits

Author SHA1 Message Date
Glenn Strauss 660d719a2a [multiple] code reuse chunkqueue_peek_data()
code reuse of chunkqueue_peek_data() and chunkqueue_read_data()
2020-10-11 12:19:27 -04:00
Glenn Strauss 29e66e70e2 [mod_magnet] magnet.attract-response-start-to

add option to run lua scripts in lighttpd response start hook

allows for response header manipulation

new params provide read-only access:

allows for content manipulation if the response body is complete

The HTTP response status can be accessed in lua via
lighty.env["response.http-status"] and should be checked, as
appropriate, prior to body manipulation.  The value is non-zero
in response start hook (magnet.attract-response-start-to), but is
likely to be 0 in scripts run from other lighttpd hooks earlier in
request processing,
e.g. magnet.attract-raw-url-to or magnet.attract-physical-path-to

Caller should check lighty.env["response.body-length"]
is a smaller and sane amount to read into memory and copy
a second time into lua data structures.  The value is lua nil
if the response body is not yet complete (or if it is >= 2GB-1)

Loading the response body (and all mod_magnet lua scripts) are
executed serially (blocking) in lighttpd, so its use is highly
discouraged on large files.  The body can be accessed in lua via
lighty.env["response.body"] if the response body is complete.
(recommended config option: = 0 (default)
 if mod_magnet scripts must process the response body)

Modifying HTTP response status and response body has not changed
and is achieved by setting lua script return value and modifying
the lighty.content lua table.

(note: mod_magnet, mod_setenv, mod_deflate, mod_expire have their
 response start hooks run in the order listed in server.modules)
2020-10-11 12:19:27 -04:00
Glenn Strauss 16a70b9253 [multiple] TLS modules use chunkqueue_peek_data() 2020-10-11 12:19:27 -04:00
Glenn Strauss 6c68e14940 [core] chunkqueue_{peek,read}_data(), squash
chunkqueue_peek_data(), chunkqueue_read_data(), chunkqueue_read_squash()
shared code for chunkqueue manipulation
2020-10-11 12:19:27 -04:00
Glenn Strauss 362be7b3bb [core] relay 1xx from backend over HTTP/1.1
relay 1xx from backend over HTTP/1.1, e.g. 103 Early Hints
(if client is connected using HTTP/1.1)

enabled by default unless disabled in lighttpd.conf with:
  server.feature-flags += ( "server.h1-discard-backend-1xx" = "enable" )

Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response.  Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request. Security Considerations

  An HTTP Status Code for Indicating Hints (103 Early Hints)
2020-10-11 12:19:27 -04:00
Glenn Strauss 869c778aa7 [core] relay 1xx from backend over HTTP/2
relay 1xx from backend over HTTP/2, e.g. 103 Early Hints
(if client is connected using HTTP/2)

enabled by default unless disabled in lighttpd.conf with:
  server.feature-flags += ( "server.h2-discard-backend-1xx" = "enable" )

Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response.  Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request. Security Considerations

  An HTTP Status Code for Indicating Hints (103 Early Hints)
2020-10-11 12:19:26 -04:00
Glenn Strauss 10d9d14633 [core] reload c after chunkqueue_compact_mem()
reload c = cq->first after calling chunkqueue_compact_mem()
2020-10-11 12:19:26 -04:00
Glenn Strauss edbfe36563 [core] support multiple 1xx responses from backend
support multiple 1xx intermediate responses from backends

Currently, all 1xx responses from backends are discarded.
In the future, these 1xx responses may be forwarded to the client
(when lighttpd also configured = 1 or = 2)
2020-10-11 12:19:26 -04:00
Glenn Strauss 7a3098398c [core] connection_check_expect_100()
isolate code to check for Expect: 100-continue
(separate out from code to read request body)
2020-10-11 12:19:26 -04:00
Glenn Strauss a330746f06 [TLS] error if inherit empty TLS cfg from globals
error if $SERVER["socket"] inherits empty TLS config from global scope
and ssl.engine = "enable" in the $SERVER["socket"]
2020-10-11 12:19:26 -04:00
Glenn Strauss 5ba175269a [core] skip ls-hpack decode work unused by lighttpd
define LSHPACK_DEC_CALC_HASH 0 to skip work whose results are not used
by lighttpd
2020-10-11 12:19:26 -04:00
Glenn Strauss 3fbb5773e9 [core] http_header_e <=> lshpack_static_hdr_idx
map enum http_header_e to enum lshpack_static_hdr_idx
map enum lshpack_static_hdr_idx to enum http_header_e
2020-10-11 12:19:26 -04:00
Glenn Strauss 367f30a645 [multiple] extend enum http_header_e list 2020-10-11 12:19:26 -04:00
Glenn Strauss 2e0676fd6d [core] extend (data_string *) to store header id
(optional addition to (data_string *), used by http_header.[ch])

extend (data_string *) instead of creating another data_* TYPE_*
  (new data type would probably have (data_string *) as base class)
  (might revisit choice in the future)

HTTP_HEADER_UNSPECIFIED has been removed.  It was used in select
locations as an optimization to avoid looking up enum header_header_e
before checking the array, but the ordering in the array now relies
on having the id.  Having the id allows for a quick check if a common
header is present or not in the htags bitmask, before checking the
array, and allows for integer comparison in the log(n) search of the
array, instead of strncasecmp().

With HTTP_HEADER_UNSPECIFIED removed, add optimization to set bit
in htags for HTTP_HEADER_OTHER when an "other" header is added,
but do not clear the bit, as there might be addtl "other" headers
2020-10-11 12:19:26 -04:00
Glenn Strauss 68ec5ad642 [core] array.[ch] using uint32_t instead of size_t 2020-10-11 12:19:26 -04:00
Glenn Strauss 92e5a021d2 [core] http_header_hkey_get_lc() for HTTP/2
header field-names are required to be lowercase for HTTP/2
so specialize http_header_hkey_get() as http_header_hkey_get_lc()
for simpler comparison

lowercase field-names in http_headers[], as it does not matter for
buffer_eq_icase_ssn(), which is used with http_headers[] for HTTP/1.x
2020-10-11 12:19:26 -04:00
Glenn Strauss babf0d0e4c [core] separate http_header_e from _htags bitmask
separate http_header_e index from r->{rqst,resp}_htags bitmask
(allows http_header_e remain 32-bit and to be used in array indexes
 while also allowing r->{rqst,resp}_htags to grow to 64-bits wide
 in the future, without requiring invasive changes)
2020-10-11 12:19:26 -04:00
Glenn Strauss 9c8981a7d1 [core] tst,set,clr macros for r->{rqst,resp}_htags 2020-10-11 12:19:26 -04:00
Glenn Strauss c58b95f297 [core] light_isupper(), light_islower()
more efficient char checks
(replace one comparision and one branch with one subtraction)
2020-10-11 12:19:26 -04:00
Glenn Strauss 327de98b38 [core] adjust response header prep for common case 2020-10-11 12:19:26 -04:00
Glenn Strauss c88d373507 [core] Content-Length in http_response_send_file() 2020-10-11 12:19:26 -04:00
Glenn Strauss c3a85c9bf5 [mod_wolfssl] standalone module
standalone module forked from mod_openssl
2020-10-11 12:19:26 -04:00
Glenn Strauss 1d9709b8ab [core] update HTTP status codes list from IANA 2020-10-11 12:19:26 -04:00
Glenn Strauss 38620b10bc [core] cold h2_log_response_header*() funcs 2020-10-11 12:19:26 -04:00
Glenn Strauss 97e314fc9e [multiple] inline chunkqueue_length() 2020-10-11 12:19:26 -04:00
Glenn Strauss 3a766d3d02 [mod_webdav] preserve bytes_out when chunks merged 2020-10-11 12:19:26 -04:00
Glenn Strauss 9f8a896891 [mod_ssi] update chunk after shell output redirect
chunkqueue_update_file() to update size of file chunk after
redirecting shell output to file fd
2020-10-11 12:19:26 -04:00
Glenn Strauss dbd458a0e6 [core] set r->state in h2.c
allow h2.c layer to modify r->state in addition to r->h2state

Do not retire/release r on error if trailers (as opposed to headers)
(stream r was just allocated in the same scope for headers,
 so ok to release)
2020-10-11 12:19:26 -04:00
Glenn Strauss 4474e6117f [core] log stream id with debug.log-state-handling
log stream id with debug.log-state-handling in

(id will be 0 for HTTP/2 connection, but not for streams on connection)
(id will be 0 for HTTP/1.x requests)
2020-10-11 12:19:26 -04:00
Glenn Strauss 05ff9c570e [core] combine Cookie request headers with ';'
(thx avij)

clients should send a single Cookie header with multiple cookie values
separated with ';'.

However, HTTP/2 loosens this requirement for Cookie.
  Section Compressing the Cookie Header Field
and some HTTP/2 clients (Chrome, Firefox) send multiple
'cookie:' headers in a HEADERS frame.
2020-10-11 12:19:26 -04:00
Glenn Strauss c946b198a3 [core] code size: disable XXH64(), XXH3()
only XXH32() is used by ls-hpack, so disable XXH64() and XXH3()
to reduce code size.  (This maybe be changed in the future if
XXH64() or XXH3() are utilitized for their performance)
2020-10-11 12:19:26 -04:00
Glenn Strauss f52a10545a [mod_openssl] clarify trace w/ deprecated options 2020-10-11 12:19:26 -04:00
Glenn Strauss f2e4e83a85 [mod_dirlisting] fix config parsing crash
(bug on master branch; never released)

(thx avij)

update doc that dir-listing.hide-dotfiles = "enable" by default
since lighttpd 1.4.40
2020-10-11 12:19:26 -04:00
Glenn Strauss 32d4569512 [core] xxhash no inline hints; let compiler choose 2020-10-11 12:19:26 -04:00
Glenn Strauss a25aeaa2f6 [build] remove ls-hpack/deps
xxHash is provided separately by lighttpd src/algo_xxhash.[ch]
2020-10-11 12:19:26 -04:00
Glenn Strauss 5ec59999dd [build] modify build, includes for xxHash v0.8.0 2020-10-11 12:19:26 -04:00
Glenn Strauss 85c1a1743e [core] import xxHash v0.8.0
import xxhash.[ch] as algo_xxhash.[ch]
tag: v0.8.0 94e5f23e736f2bb67ebdf90727353e65344f9fc0
2020-10-11 12:19:26 -04:00
Glenn Strauss 156e1da2ae [core] rename splaytree.[ch] to algo_splaytree.[ch] 2020-10-11 12:19:26 -04:00
Glenn Strauss 3ffb195532 [core] move djbhash(), dekhash() to algo_md.h 2020-10-11 12:19:26 -04:00
Glenn Strauss 6c8160e5a4 [core] rename md5.[ch] to algo_md5.[ch] 2020-10-11 12:19:26 -04:00
Glenn Strauss 044c8e80e5 [core] use djbhash in gw_backend to choose host
djbhash() is faster and more cache-friendly than is crc32c()
2020-10-11 12:19:26 -04:00
Glenn Strauss a3af9833c6 [core] fix crash on master if blank line request
(bug on master branch; never released)

(thx avij)

fix crash on master if blank line precedes HTTP/1.1 keep-alive request

header parsing code previously made assumptions that request was
HTTP/1.0 or HTTP/1.1, where a request-line was required, and which
would error out elsewhere if request-line was missing.  The parsing
code also previously looked for "\r\n\r\n" to end headers.

The header offset parsing code was modified and invalidated the above
assumptions, now looking only for blank line "\r\n", but the calling
code had not properly been updated.  (until this patch)
2020-10-11 12:19:26 -04:00
Glenn Strauss 5a694281da [core] ls-hpack optimizations
  lighttpd does not require HTTP/1.1 output compat from HPACK decoder
  ("field-name: value\r\n")

define NDEBUG (in ls-hpack/lshpack.c)
  lighttpd spends upwards of 20% total lighttpd CPU time in HPACK
  encode/decode in h2load test on static file over cleartext (not TLS)
  Defining NDEBUG eliminates some asserts() and results in a small
  but measurable reduction in CPU usage
2020-10-11 12:19:26 -04:00
Glenn Strauss 4656d91889 [core] document algorithm used in lighttpd etag
Donald E. Knuth
The Art Of Computer Programming Volume 3
Chapter 6.4, Topic: Sorting and Search
2020-10-11 12:19:26 -04:00
Glenn Strauss 5547530a01 [core] do not require '\0' term for k,v hdr parse
no longer require '\0' terminated z-string for key,value header parsing
2020-10-11 12:19:26 -04:00
Glenn Strauss 5fd8a26a75 [core] defer optimization to read small files
defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured
2020-10-11 12:19:24 -04:00
Glenn Strauss 3baef447b3 [core] fix crash on master after graceful restart
(bug on master branch; never released)

(thx nate)
2020-10-11 11:43:06 -04:00
Glenn Strauss d8b88be056 [core] move connections_get_state to connections.c
connections_get_state() is used only in connections.c for debugging
2020-10-11 11:43:06 -04:00
Glenn Strauss 0f9ec8b755 [mod_status] dedicated funcs for r->state labels
fork connection_get_state() for use by mod_status, which
might in the future choose to display different labels.
move and rename connection_get_short_state() into mod_status,
as the func is used only by mod_status

(connection_get_state() is currently used elsewhere only for debugging
 in connections.c)
2020-10-11 11:43:06 -04:00
Glenn Strauss 878073d1c9 [core] keep pool of (request_st *) for HTTP/2 2020-10-11 11:43:06 -04:00
Glenn Strauss 550609c8ae [core] request_acquire(), request_release()
continue code abstraction for (request_st *)
2020-10-11 11:43:06 -04:00
Glenn Strauss 0905b6f7f9 [core] fix 1-char trunc of default server.tag
(thx avij)
2020-10-11 11:43:06 -04:00
Glenn Strauss cf0098eac8 [mod_auth] fix crash if auth.require misconfigured (fixes #3023)
(thx veyrdite)

  "Segfault with mod_auth & htpasswd (lighttpd.conf misconfig)"
2020-10-11 11:43:06 -04:00
Glenn Strauss cae205ad99 [build] fix meson build 2020-10-11 11:43:06 -04:00
Glenn Strauss 79e2f5baee [core] ignore empty field-name from backends
Empty field-name is already ignored when generating response headers,
but this is an explicit skip of empty field-name sent from backends,
including field-names that look like HTTP/2 pseudo-headers, as those
begin with ':'.
2020-10-11 11:43:06 -04:00
Glenn Strauss 8e2cf55250 [core] redirect to dir using relative-path
Location response header is permitted to use relative-path in
RFC 7231 Section 7.1.2. Location

Prefer relative path in redirection for the benefit of reverse proxies
and CDNs.  Doing so also avoids potentially disclosing internal schemes
and server names which client might not be able to directly reach.

To restore prior behavior of sending a fully-qualified absolute URI:
server.feature-flags += ("absolute-dir-redirect" => "enable")

2020-10-11 11:43:06 -04:00
Glenn Strauss baa651a3ab [mod_status] adjust to print HTTP/2 requests 2020-10-11 11:43:06 -04:00
Glenn Strauss 9d74aa600b [mod_status] separate funcs to print request table 2020-10-11 11:43:06 -04:00
Glenn Strauss 06df2716f2 [core] comment possible future ls-hpack optimize 2020-10-11 11:43:06 -04:00
Glenn Strauss 8fc8ab891a [core] http_request_parse_header() specialized
http_request_parse_header() specialized for HTTP/2 request headers
to be parsed as each field-name and value is HPACK-decoded; send headers
directly from HPACK decoder, rather than double-buffering in chunkqueue

http_request_headers_process_h2() for post-processing
2020-10-11 11:43:06 -04:00
Glenn Strauss ada09a23b0 [core] h2_send_headers() specialized for resp hdrs
specialized version of http_response_write_header(); send headers
directly to HPACK encoder, rather than double-buffering in chunkqueue
2020-10-11 11:43:06 -04:00
Glenn Strauss 014e5240ef [core] HTTP/2 HPACK using LiteSpeed ls-hpack
2020-10-11 11:43:06 -04:00
Glenn Strauss 70b1af0640 [core] link in ls-hpack (EXPERIMENTAL)
LiteSpeed ls-hpack v2.2.1

XXX: might be better to include this as a git submodule
but minor code changes were made here for portability:
- C99 flexible array members defined as a[] instead of a[0])
- pedantic compiler warnings (excess ';' and missing declarations)
- deletion of large tables from ls-hpack/huff-tables.h (code size)
2020-10-11 11:43:06 -04:00
Glenn Strauss 1bf4555800 [core] update comment 2020-10-11 11:43:06 -04:00
Glenn Strauss 501182dc4b [core] adjust (disabled) debug code 2020-10-11 11:43:06 -04:00
Glenn Strauss 1c7c19a9d5 [core] skip write throttle for 100 Continue
(small code simplification)
2020-10-11 11:43:06 -04:00
Glenn Strauss 6fc8ad3194 [core] discard 100 102 103 responses from backend 2020-10-11 11:43:06 -04:00
Glenn Strauss 4d9c54f61b [core] mark some error paths with attribute cold 2020-10-11 11:43:06 -04:00
Glenn Strauss 8d7e9439b5 [core] connection transition to HTTP/2; incomplete

(basic framework with some stub functions; incomplete)
(subsequently incrementally updated using git rebase)

HTTP/2 via TLS ALPN extension  (TLS)
HTTP/2 via Upgrade: h2c        (cleartext)
HTTP/2 via Prior Knowledge     (cleartext)

server.feature-flags += ("server.h2proto" => "enable")
to enable HTTP/2
server.feature-flags += ("server.h2c" => "enable")
to enable upgrade to HTTP/2 over cleartext http
2020-10-11 11:43:06 -04:00
Glenn Strauss 784f1ac1da [core] alternate between two joblists
process one joblist and queue work up on second joblist

loop to poll, but poll immediate if work was queued on second joblist
2020-10-11 11:43:06 -04:00
Glenn Strauss 6eca864c5e [core] h2.[ch] with stub funcs (incomplete)
(subsequently incrementally updated using git rebase)

huge props and many thank yous to writers of testing tools used while
developing HTTP/2 support in lighttpd:

  h2spec - conformance testing tool for HTTP/2 implementation
  h2load - HTTP/2 benchmarking tool
  curl   - command line tool and library for transferring data with URLs
2020-10-11 11:43:02 -04:00
Glenn Strauss 2c8f1b4621 [core] isolate more resp code in response.c
http_response_handler() and response generation flow control logic
2020-10-03 09:05:39 -04:00
Glenn Strauss 2f2eec18fb [multiple] isolate more con code in connections.c
move code from connections-glue.c back into connections.c

move code from connections-glue.c to http-header-glue.c
  rename connection_response_reset()
      to http_response_reset()
  rename connection_handle_read_post_error()
      to http_response_reqbody_read_error()
2020-10-03 09:05:38 -04:00
Glenn Strauss 8940fec894 [multiple] modules read reqbody via fn ptr
r->con->reqbody_read() replaces connection_handle_read_post_state()

future: might provide different callbacks for request body with
Content-Length versus request body sent via Transfer-Encoding: chunked
2020-10-03 09:05:38 -04:00
Glenn Strauss 4e4026703d [core] reqpool.[ch] for (request_st *)
move request_init() request_reset() request_free() from connections.c
2020-10-03 09:05:38 -04:00
Glenn Strauss 6971c6c811 [core] move http_request_headers_process()
move http_request_headers_process() to request.[ch]
2020-10-03 09:05:38 -04:00
Glenn Strauss db7b51a4f6 [core] http_header_parse_hoff()
moved and renamed from connections.c:connection_read_header_hoff()
2020-10-03 09:05:38 -04:00
Glenn Strauss 89b3d73daa [core] adjust http_request_headers_process()
adjust for code reuse, and moving routine to lower-level file
2020-10-03 09:05:38 -04:00
Glenn Strauss 517e32785e [core] HTTP2-Settings 2020-10-03 09:05:38 -04:00
Glenn Strauss 4e656f5354 [core] connection_set_fdevent_interest()
extracted from connection_state_machine() (for reuse)

include interest for POLLRDHUP unless already received
2020-10-03 09:05:38 -04:00
Glenn Strauss d4937e29f1 [mod_accesslog,mod_rrdtool] HTTP/2 basic accounting
Note: rrdtool counts do not include HTTP/2 protocol overhead.
Continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
2020-10-03 09:05:38 -04:00
Glenn Strauss afc2025d8e [core] reset connection counters per connection
reset connection counters per connection, not per request

adjust mod_accesslog and mod_rrdtool usage

continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
2020-10-03 09:05:38 -04:00
Glenn Strauss 8eea3bd014 [core] connection_state_machine_loop()
extracted from connection_state_machine() (for reuse)
2020-10-03 09:05:38 -04:00
Glenn Strauss 68dd6312a2 [core] request_headers_process()
extract code from connection_handle_read_state() for future reuse
2020-10-03 09:05:38 -04:00
Glenn Strauss f37c90bccc [core] parse h2 pseudo-headers 2020-10-03 09:05:38 -04:00
Glenn Strauss 4c699be6c6 [core] recognize status: 421 Misdirected Request 2020-10-03 09:05:38 -04:00
Glenn Strauss 77057a7c02 [core] lowercase response hdr field names for h2 2020-10-03 09:05:38 -04:00
Glenn Strauss e089facf84 [core] do not send Connection: close if h2 2020-10-03 09:05:38 -04:00
Glenn Strauss 21987c863f [mod_extforward] preserve changed addr for h2 con
Preserve changed addr for lifetime of h2 connection; upstream proxy
should not reuse same h2 connection for requests from different clients
2020-10-03 09:05:38 -04:00
Glenn Strauss bbcc2f229a [multiple] allow TLS ALPN "h2" if "server.h2proto" 2020-10-03 09:05:38 -04:00
Glenn Strauss a8f8d5edc0 [core] HTTP_VERSION_2 2020-08-13 15:05:25 -04:00
Glenn Strauss 1041a56dd9 [core] server.feature-flags to enable h2
server.feature-flags = ("server.h2proto" => "enable") to enable h2
2020-08-13 15:05:25 -04:00
Glenn Strauss 96324fbba0 [mod_extforward] skip after HANDLER_COMEBACK
do not re-run mod_extforward uri handler after HANDLER_COMEBACK

add some additional comments
2020-08-13 15:05:25 -04:00
Glenn Strauss 0c7d250034 [mod_extforward] save proto per connection
Even though request headers are per-request, update the proto on the
connection level for trusted (proxy) clients.  Note: the proxy must use
each connection only for a single connection from a single client
(typically true in practice), and not for multiple clients.

proto was previously saved on connection level for mod_extforward with
HAProxy PROXY protocol, but did not occur with X-Forwarded-Proto or the
Forwarded request header with proto=...

Before this change, modules which returned HANDLER_COMEBACK could lose
HTTPS=on if the resulting request was for a backend such as mod_fastcgi.
This was reported in mod_rewrite, but could also affect mod_magnet if
MAGNET_RESTART_REQUEST, or mod_cgi with cgi.local-redir = "enable"

  "FastCGI behavior different when using rewrite?"
2020-08-13 13:05:12 -04:00
Glenn Strauss b600b75f20 [core] inline buffer_reset()
buffer_reset() is used on pre-allocated buffers, so remove NULL check
2020-08-10 20:05:02 -04:00
Glenn Strauss e1bb579361 [core] disperse settings.h to appropriate headers 2020-08-10 20:04:57 -04:00
Glenn Strauss 3e046ccabf [core] dlsym for FAMNoExists() for compat w/ fam
gamin should be used instead of fam; fam is no longer maintained

This patch makes it safe to build lighttpd with gamin, but run on
systems with the (deprecated) fam installed, which can happen due
to historical package dependency declarations on some platforms
(Debian, Ubuntu).

gamin and fam are not 100% binary compatible.
(Among other things, fam does not provide FAMNoExists())

  "llibgamin vs libfam conflict solving"
  "libgamin0: libfam shlib dependency wrongly set to libfam0"
  "undefined symbol: FAMNoExists"
2020-08-10 15:56:28 -04:00
Glenn Strauss d2d5f27f80 [core] chunkqueue_compact_mem() must upd cq->last 2020-08-10 12:54:33 -04:00
Glenn Strauss 346280fed7 [mod_proxy] do not forward Expect: 100-continue
do not forward Expect: 100-continue to backend
since we do not handle HTTP/1.1 100 Continue response
2020-08-10 12:54:33 -04:00
Glenn Strauss 8b4abaf498 [mod_webdav] fix fallback if linkat() fails
fix fallback if linkat() fails

check at startup if /proc/self/fd is present on systems with O_TMPFILE
(containers might not mount /proc)

  "mod_webdav - PUT files with < 64kb Content-Length reults in zero length file"
2020-08-10 12:54:33 -04:00
Glenn Strauss eb4f953396 [build] avoid accept_filter_arg compiler warning
avoid compiler warning for struct accept_filter_arg af_name strncpy()
2020-08-02 12:55:18 -04:00
Glenn Strauss f358f04bdc [build] SCons build mod_deflate w/ libm for brotli 2020-08-02 12:32:37 -04:00
Glenn Strauss 4663564c8a [mod_authn_dbi,mod_authn_mysql] fix coverity issue 2020-08-02 09:00:17 -04:00
Glenn Strauss 7bc7082d32 [mod_extforward] attempt to quiet Coverity warning 2020-08-02 09:00:17 -04:00
Glenn Strauss 4a1960958a [core] add some missing standard includes 2020-08-02 07:47:42 -04:00
Glenn Strauss b2d7635801 [mod_webdav] fix inadvertent string truncation
(bug on master branch; never released)
2020-08-02 07:47:42 -04:00
Glenn Strauss 0fb391c096 [core] http_header_remove_token() 2020-08-02 07:47:42 -04:00
Glenn Strauss d44a26bfa3 [tests] test_base64.c clear buf vs reset
(can be slightly faster to clear buffer than to reset buffer)
2020-08-02 07:47:42 -04:00
Glenn Strauss f7919c1ae3 [mod_proxy] send HTTP/1.1 requests to backends
For prior behavior (HTTP/1.0 requests to backend), force HTTP/1.0 with:
  server.feature-flags = ("proxy.force-http10" => "enable")
2020-08-02 07:47:42 -04:00
Glenn Strauss c3073b4da6 [core] stricter parsing of POST chunked block hdr 2020-08-02 07:47:42 -04:00
Glenn Strauss 9ca668c870 [mod_fastcgi] decode Transfer-Encoding: chunked
decode Transfer-Encoding: chunked from mod_fastcgi backend
2020-08-02 07:47:42 -04:00
Glenn Strauss 7420526ddb [core] decode Transfer-Encoding: chunked from gw
decode Transfer-Encoding: chunked from gw (gateway backends)

Transfer-Encoding: chunked is a hop-by-hop header.

Handling chunked encoding remove a hurdle for mod_proxy to send HTTP/1.1
requests to backends and be able to handle HTTP/1.1 responses.

Other backends ought not to send Transfer-Encoding: chunked, but in
practice, some implementations do.
2020-08-02 07:47:42 -04:00
Glenn Strauss dbd438cca0 [mod_openssl] read secret data w/ BIO_new_mem_buf
Using BIO_new_mem_buf() allows lighttpd to control file reads and
securely wiping memory, avoiding the use of BIO_s_file(), which
employs stdio and BIO internal copying without securely wiping memory.

BIO_new_mem_buf() is also much more performant than BIO_s_file()
or BIO_new_fd() without a buffering BIO, which might make many
syscalls reading a single character at a time.

BIO_new_mem_buf() appears to be supported by older OpenSSL versions
as well as by numerous OpenSSL-compatible APIs, e.g. in BoringSSL,
LibreSSL, WolfSSL
2020-08-02 07:47:42 -04:00
Glenn Strauss 4b755d2b03 [mod_deflate] quicker check for Content-Encoding
The response is not a candidate for mod_deflate if
Content-Encoding was already set by something else.
2020-08-02 07:47:42 -04:00
Glenn Strauss 496dc6d9c5 [core] detect and reject TLS connect to cleartext
detect and reject TLS connection to cleartext listening port
(alternative to failing to receive HTTP header and waiting to time out)
2020-08-02 07:47:42 -04:00
Glenn Strauss 87428818a1 [mod_nss] func renames for consistency
mod_nss was written using mod_mbedtls and mod_gnutls as templates,
but some code was accidentally committed with those other modules'
2020-08-02 07:47:41 -04:00
Glenn Strauss 33c8cf41db [multiple] rename connection_reset hook to request
rename connection_reset to handle_request_reset
2020-08-02 07:47:41 -04:00
Glenn Strauss bb6a5d7821 [core] separate funcs to reset (request_st *)
request_init(), request_free(), request_reset() separate from (and
called by) connection_init(), connection_free(), connection_reset()
2020-08-02 07:47:41 -04:00
Glenn Strauss 164f7600b7 [multiple] con hooks store ctx in con->plugin_ctx
modules with connection level hooks now store ctx in con->plugin_ctx
2020-08-02 07:47:41 -04:00
Glenn Strauss fea5bdc873 [core] con layer plugin_ctx separate from request 2020-08-02 07:47:41 -04:00
Glenn Strauss 5f7bf472c7 [core] server.feature-flags extensible config
server.feature-flags = ("server.xxxx" => "enable")
2020-08-02 07:46:27 -04:00
Glenn Strauss bc5d4b3492 [mod_compress] remove mod_compress 2020-08-02 06:46:22 -04:00
Glenn Strauss 8dd33a72dd [mod_deflate] mod_deflate subsumes mod_compress
translate config server.modules "mod_compress" to "mod_deflate"
accept compress.* directives, but issue DEPRECATED warning trace

mod_deflate differences from mod_compress:
- mod_compress compress.filetype was exact match; deflate.mimetypes is
  prefix match (behavior change might compress longer mimetype matches,
  which are likely of similar type and compressability)
- mod_compress always sent entire (compressed) file for Range request
  mod_deflate will stream compress range result (not stored in cache)
- mod_compress would short-circuit request with 403 Forbidden error
  if request file did not exist (stat() failed) (This behavior was
  unfriendly to other handlers)
- mod_compress compress.cache-dir layout differs from deflate.cache-dir
  layout; file cache should be cleared (or renamed) when migrating from
  mod_compress to mod_deflate
- mod_deflate does not issue Vary: Accept-Encoding if request does not
  contain Accept-Encoding.  The identity response can be cache by
  proxies and served to clients.  Historically, some proxies disabled
  caching if any Vary: response was seen.  If the Vary header is
  desirable, mod_deflate code which checks for Accept-Encoding and
  compression type can be moved down a few lines to be below the
  setting of the Vary response header.
2020-08-02 06:46:22 -04:00
Glenn Strauss be6da785c5 [mod_deflate] deflate.cache-dir compressed cache
(achieves feature parity with mod_compress)
2020-08-02 06:46:22 -04:00
Glenn Strauss 5b15c1f449 [mod_deflate] propagate errs from internal funcs
also, pass one less arg to internal funcs
2020-08-02 06:46:22 -04:00
Glenn Strauss b2fe37f0a9 [mod_deflate] fix typo in config option
(typo on master branch; never released)
2020-08-02 06:46:22 -04:00
Glenn Strauss 8d5e237c60 [mod_deflate] Brotli support
configuration option: ./configure --with-brotli
2020-08-02 06:46:22 -04:00
Glenn Strauss be8047d82f [build] build fixes for AIX
  "Compile on AIX conflicting types for 'etag_create'"
2020-07-21 12:08:27 -04:00
Glenn Strauss 1a516b4c4c [mod_vhostdb_dbi] allow strings and ints in config
e.g. allow "port" value to be a string or integer instead of requiring
     port be a string
2020-07-16 00:30:20 -04:00
Glenn Strauss 3dca923591 [mod_authn_mysql,file] use crypt() to save stack
use crypt() instead of crypt_r() to save stack space,
as struct crypt_data might be very large.

While crypt() is not thread-safe, lighttpd is single-threaded
2020-07-16 00:29:43 -04:00
Glenn Strauss fed2ecae19 [mod_authn_dbi] authn backend employing DBI 2020-07-16 00:29:43 -04:00
Glenn Strauss dba4a328c2 [core] Nettle assert()s if buffer len > digest sz 2020-07-13 17:39:30 -04:00
Glenn Strauss c3dd167dd9 [mod_openssl] fix build warnings on MacOS X
need undef OPENSSL_NO_STDIO for a few funcs which access filesystem
2020-07-13 17:39:30 -04:00
Glenn Strauss 33ae830291 [mod_auth] accept "nonce-secret" & "nonce_secret" 2020-07-13 17:39:30 -04:00
Glenn Strauss 563fe5f013 [mod_authn_ldap,mod_vhostdb_ldap] add timeout opt (#2805)
auth.backend.ldap.timeout = "2000000"    # quoted-string; microseconds
vhostdb.ldap += ("timeout" => "2000000") # quoted-string; microseconds

Default is 2000000 microseconds (2 secs)

These values are converted to struct timeval and passed to
  ldap_set_option(ld, LDAP_OPT_NETWORK_TIMEOUT, );
  ldap_set_option(ld, LDAP_OPT_TIMEOUT, ...);
if those LDAP_OPT_* values are available (both are OpenLDAP-specific).

  "mod_auth caching"
2020-07-13 17:39:30 -04:00
Glenn Strauss e11514b086 [mod_auth,mod_vhostdb] add caching option (fixes #2805)
auth.cache    = ("max-age" => "600")
vhostdb.cache = ("max-age" => "600")

If specified with an empty array, default max-age is 600 secs (10 mins)
auth.cache    = ()
vhostdb.cache = ()

(Note: cache expiration occurs every 8 seconds, so maximum cache time
 might be up to max-age + 8 seconds)

  "mod_auth caching"
2020-07-13 17:39:27 -04:00
Rosen Penev fd2a12d636 [meson] add missing libmaxminddb section
Fixes the following error when building with -Dwith_maxminddb=true: ERROR: Unknown variable "libmaxminddb".

A full log can be found at meson-logs/meson-log.txt

Signed-off-by: Rosen Penev <>
2020-07-11 01:07:19 -04:00
Rosen Penev 04a7d98cb9 [meson] fix libmariadb dependency
libmariadb is what should be used as only the library portion is used.

Fixes compilation under OpenWrt.

Note that mariadb.pc is a superset that links to libmariadb.

Signed-off-by: Rosen Penev <>
2020-07-11 01:07:16 -04:00
Glenn Strauss 3057b63ea2 [multiple] address coverity warnings 2020-07-10 22:18:26 -04:00
Glenn Strauss 3119d938cb [mod_webdav] build fix: ifdef live_properties 2020-07-10 21:54:27 -04:00
Glenn Strauss 50ddc8108b [mod_webdav] update rc for PROPFIND allprop
(rc currently ignored in case of PROPFIND allprop)
2020-07-10 21:36:32 -04:00
Glenn Strauss 10b7aedcdb [mod_webdav] fix theoretical NULL dereference
fix theoretical NULL dereference identified by Coverity Scan

possible for PROPFIND with specific atypical choices in lighttpd.conf:
- possible for getcontenttype if no content type matches resource
  and no default type configured in lighttpd.conf
- possible for getetag if etag disabled in lighttpd.conf
2020-07-10 21:34:28 -04:00
Glenn Strauss 2781a3be6d [multiple] address coverity warnings 2020-07-10 21:34:28 -04:00
Glenn Strauss ea58cc6f3f [core] store fd in srv->stdin_fd during setup
(pedantic; no impact)

upon error, server will exit, so the impact of momentarily leaking fd
has no impact.  This commit holds the fd in srv->stdin_fd to address
Coverity warning about leaking fd when using server.bind = "/dev/stdin"
2020-07-10 19:30:24 -04:00
Glenn Strauss 6886e78bca [core] remove unused mbedtls_enabled flag
(initially used during development of mod_mbedtls, but no longer used)
2020-07-10 19:28:12 -04:00
Glenn Strauss 2f5fc68b59 [build] fix typo in src/ EXTRA_DIST 2020-07-10 19:27:42 -04:00
Glenn Strauss be0d11414e [cmake] update deps for src/t/test_* 2020-07-10 03:20:37 -04:00
Glenn Strauss 52fb78459f [cmake] update deps for src/t/test_* 2020-07-10 03:13:08 -04:00
Glenn Strauss f85d1f9070 [core] splaytree_djbhash() in splaytree.h (reuse)
rename and move hashme() from stat_cache.c to splaytree_djbhash() in
splaytree.h for code reuse
2020-07-10 00:02:48 -04:00
Glenn Strauss 0d4c8edd8f [mod_mysql_vhost] deprecated; use mod_vhostdb_mysql
add warning at server startup when mod_mysql_vhost is loaded

mod_vhostdb_mysql subsumes mod_mysql_vhost.  Individual mod_mysql_vhost
directives map one-to-one to keywords in vhostdb.mysql = (...) directive
2020-07-09 23:45:06 -04:00