Commit Graph

3074 Commits

Author SHA1 Message Date
Glenn Strauss 6d76f7ce49 [core] fix chunkqueue_compact_mem w/ partial chunk
(bug on master branch; never released)
2020-10-19 21:40:14 -04:00
Glenn Strauss 8abbf621d1 [mod_fastcgi] decode chunked is cold code path
decode chunked from FastCGI backend is cold code path
2020-10-19 21:40:14 -04:00
Glenn Strauss 96abd9cfb8 [core] coalesce nearby ranges in Range requests
Range requests must be HTTP/1.1 or later (not HTTP/1.0)
2020-10-19 21:40:14 -04:00
Glenn Strauss 66d1ec485c [core,mod_deflate] leverage cache of open fd
leverage simple cache of open file in stat_cache
(use stat_cache_get_entry_open())

future: reference count fd instead of dup()
  (requires extending chunkqueue interfaces)
2020-10-19 21:40:14 -04:00
Glenn Strauss d8e5e21eb7 [core] stat_cache_get_entry_open()
simple interface to cache open file by extending struct stat_cache_entry

future: should probably create fd cache separate from stat_cache,
        perhaps along w/ http-specific fields like etag and content_type
2020-10-19 21:40:14 -04:00
Glenn Strauss e99126074c [core] pass open fd to http_response_parse_range 2020-10-13 22:31:10 -04:00
Glenn Strauss 6219b861ce [core] http_response_parse_range() const file sz 2020-10-13 22:31:10 -04:00
Glenn Strauss cd2ff4202d [core] remove unused r->content_length
(was parsed Content-Length sent by backend)
(might add back in future along with stricter validation of field value)
2020-10-13 22:31:10 -04:00
Glenn Strauss 55e9f082a9 [core] limit num ranges in Range requests
parse Range header prior to constructing ranged response
2020-10-13 22:31:10 -04:00
Glenn Strauss 2dc3d5faac [mod_gnutls] quiet CLOSE_NOTIFY after conn reset
do not log error after connection reset
2020-10-13 22:31:10 -04:00
Glenn Strauss 496cd8ff44 [mod_mbedtls] quiet CLOSE_NOTIFY after conn reset
do not log error after connection reset
2020-10-13 22:31:10 -04:00
Glenn Strauss f846a392d5 [multiple] use stat_cache_path_isdir() 2020-10-13 22:31:10 -04:00
Glenn Strauss 7d368cd7a5 [core] stat_cache_path_isdir() 2020-10-13 22:31:07 -04:00
Glenn Strauss 14eb97f1b3 [core] dup FILE_CHUNK fd when splitting FILE_CHUNK 2020-10-11 12:19:27 -04:00
Glenn Strauss 2b4166fb74 [core] fdevent_dup_cloexec() 2020-10-11 12:19:27 -04:00
Glenn Strauss ad22eebd23 [mod_ssi] use stat_cache_open_rdonly_fstat() 2020-10-11 12:19:27 -04:00
Glenn Strauss b7370a6d46 [core] save errno around close() if fstat() fails
(should not happen, since file was just opened)
2020-10-11 12:19:27 -04:00
Glenn Strauss e2c832122b [mod_deflate] fix potential NULL deref in err case
(bug on master branch; never released)
2020-10-11 12:19:27 -04:00
Glenn Strauss 4d6d1e790a [multiple] use light_btst() for hdr existence chk 2020-10-11 12:19:27 -04:00
Glenn Strauss bd8edb51d0 [core] allow symlinks under /dev for rand devices
(fix code to match comment)
2020-10-11 12:19:27 -04:00
Glenn Strauss 874707cd66 [TLS] use fdevent_load_file_bytes() for STEK file
remove direct dependency on <unistd.h> from lighttpd TLS modules
2020-10-11 12:19:27 -04:00
Glenn Strauss 771ebd39a9 [core] fdevent_load_file_bytes()
wrapper to load defined number of bytes from file at given offset
2020-10-11 12:19:27 -04:00
Glenn Strauss ee1f7af24d [core] be more precise checking streaming flags 2020-10-11 12:19:27 -04:00
Glenn Strauss f7fc8d8d75 [core] connection_joblist global
con->srv->joblist is expensive to traverse when cache is cold
and called from connection_handle_fdevent(); var made a global
2020-10-11 12:19:27 -04:00
Glenn Strauss 01f90c58ef [core] walk queue in connection_write_chunkqueue()
walk chunkqueue up to first FILE_CHUNK (if present)
This may incur memory load misses for pointer chasing, but effectively
preloads part of the chunkqueue, something which used to be a side
effect of a previous (less efficient) version of chunkqueue_length()
which walked the entire chunkqueue (on each and every call).  The loads
here make a measurable difference in performance in underlying call to
con->network_write()
2020-10-11 12:19:27 -04:00
Glenn Strauss f1136f7897 [core] short-circuit connection_write_throttle()
short-circuit connection_write_throttle() when throttling not enabled
2020-10-11 12:19:27 -04:00
Glenn Strauss 7c1e81299f [core] handle unexpected EOF reading FILE_CHUNK
(replace existing check which suffered from ToC-ToU race condition)
enhances logic from 2015 commit 593599f1 and avoids repeated fstat()
checks when sending large files

For mmap(), lighttpd catches SIGBUS if file is (externally) truncated
and lighttpd attempts to access bytes in a read-only mapping more than
a memory page boundary following the end of the file.

For sendfile(), lighttpd returns an error if sendfile() reports no error
and that no bytes have been sent after lighttpd attempts to send a
non-zero number of bytes.
2020-10-11 12:19:27 -04:00
Glenn Strauss a8398e4596 [core] defer handling FDEVENT_HUP and FDEVENT_ERR
defer handling FDEVENT_HUP and FDEVENT_ERR to after processing
(connection *) in order to have a chance to read data in kernel
socket buffers
2020-10-11 12:19:27 -04:00
Glenn Strauss 52d9b0da88 [core] server_run_con_queue()
func to run queued (connection *) jobs
2020-10-11 12:19:27 -04:00
Glenn Strauss a3001b968f [core] propagate state after writing
(perf) avoid reload miss after partial write of chunkqueue data
2020-10-11 12:19:27 -04:00
Glenn Strauss 81029b8b51 [multiple] inline chunkqueue where always alloc'd
inline struct chunkqueue where always allocated in other structs

(memory locality)
2020-10-11 12:19:27 -04:00
Glenn Strauss 8b382a81c7 [multiple] use sock_addr_get_family in more places 2020-10-11 12:19:27 -04:00
Glenn Strauss 86e5f09062 [core] perf adjustments to avoid load miss 2020-10-11 12:19:27 -04:00
Glenn Strauss ed297e7e67 [core] lighttpd -1 supports pipes (e.g. netcat)
lighttpd -1 (one-shot mode) now supports pipes (e.g. with netcat)

x-ref:
  "Is there anyway to run lighttpd in the command line?"
  https://redmine.lighttpd.net/boards/2/topics/9348
2020-10-11 12:19:27 -04:00
Glenn Strauss 94c4c63773 [core] graceful shutdown timeout option
server.feature-flags += ("server.graceful-shutdown-timeout" => 10)

After receiving SIGINT or SIGUSR1, lighttpd will gracefully shutdown,
waiting for existing connections to complete.  In the case of SIGUSR1,
this wait occurs before restarting lighttpd.  The default timeout is
none (unlimited).

When "server.graceful-shutdown-timeout" option is set, it defines the
number of seconds that lighttpd will wait for existing connections to
complete before shutting down the connection.

Sites which expect large uploads or downloads, or those with very slow
clients, might want to set a much longer timeout, e.g 60 seconds

For more immediate graceful restarts, while still allowing existing
connections time to complete, sites should additionally consider
whether or not
  server.feature-flags += ("server.graceful-restart-bg" => "enable")
is appropriate and compatible with their lighttpd.conf settings
2020-10-11 12:19:27 -04:00
Glenn Strauss 76bd8bba9a [mod_ssi] init status var before waitpid() 2020-10-11 12:19:27 -04:00
Glenn Strauss 352d5d776d [core] graceful and immediate restart option
graceful and (nearly) immediate lighttpd restart option

For *some* configurations, it *may* be safe to background the current
lighttpd server (or workers) to continue processing active requests
and, in parallel, to start up a new lighttpd server with a new
configuration.  For other configurations, doing so might not be safe!

Therefore, this option must be explicitly configured to enable:
  server.feature-flags += ("server.graceful-restart-bg" => "enable")
  server.systemd-socket-activation = "enable"

Along with enabling server.feature-flags "server.graceful-restart-bg",
enabling server.systemd-socket-activation allows transfer of open
listening sockets to the new lighttpd server instance, and occurs
without closing the listening sockets and without destroying the
kernel listen backlog queue on the socket.

Safe configurations may include lighttpd.conf which connect to
standalone backend daemons, e.g. proxying to other servers,
including PHP-FPM backends.

Unsafe configurations include lighttpd.conf which use "bin-path" option
in *.server configs, instructing lighttpd to execute the backends.
Using the graceful-and-immediate-restart option is likely *unsafe* if
the backend daemon expects only one instance of itself to run at a time.

Current implementation of graceful and immediate restart option keeps
the backgrounded lighttpd in the same process group, so that subsequent
SIGINT or SIGTERM will shut down both the new and the backgrounded
servers.  (An alternative option (commented out in the code) is to
background and detach from the new lighttpd process.)  Regardless,
existing subprocesses, such as CGI, remain in original process group.
As a result, the new lighttpd server may receive SIGCHLD for unknown
processes inherited from the old server, which the new lighttpd server
will reap and discard.  The original lighttpd server, now a child, will
be unable to detect exit or reap and report status on those pre-existing
subprocesses.

Graceful restart is triggered in lighttpd by sending lighttpd SIGUSR1.
If lighttpd is configured with workers, then SIGINT (not SIGUSR1) is
sent to the process group, including other processes started by
lighttpd, e.g. CGI.  To work well with graceful restart, CGI scripts and
other processes should trap SIGINT (and SIGUSR1 for good measure).
Long-running scripts may want to checkpoint and close, e.g. a CGI script
implementing a long-running websocket connection.
2020-10-11 12:19:27 -04:00
Glenn Strauss f779d354a2 [core] config_plugin_value_tobool() accept "0","1" 2020-10-11 12:19:27 -04:00
Glenn Strauss 1a64c9e2f7 [core] reuse r->start_hp.tv_sec for r->start_ts
(remove duplicated field from (request_st *))
2020-10-11 12:19:27 -04:00
Glenn Strauss 660d719a2a [multiple] code reuse chunkqueue_peek_data()
code reuse of chunkqueue_peek_data() and chunkqueue_read_data()
2020-10-11 12:19:27 -04:00
Glenn Strauss 29e66e70e2 [mod_magnet] magnet.attract-response-start-to
(experimental)

add option to run lua scripts in lighttpd response start hook

allows for response header manipulation

new params provide read-only access:
  lighty.env["response.http-status"]
  lighty.env["response.body-length"]
  lighty.env["response.body"]

allows for content manipulation if the response body is complete

The HTTP response status can be accessed in lua via
lighty.env["response.http-status"] and should be checked, as
appropriate, prior to body manipulation.  The value is non-zero
in response start hook (magnet.attract-response-start-to), but is
likely to be 0 in scripts run from other lighttpd hooks earlier in
request processing,
e.g. magnet.attract-raw-url-to or magnet.attract-physical-path-to

Caller should check lighty.env["response.body-length"]
is a smaller and sane amount to read into memory and copy
a second time into lua data structures.  The value is lua nil
if the response body is not yet complete (or if it is >= 2GB-1)

Loading the response body (and all mod_magnet lua scripts) are
executed serially (blocking) in lighttpd, so its use is highly
discouraged on large files.  The body can be accessed in lua via
lighty.env["response.body"] if the response body is complete.
(recommended config option: server.stream-response-body = 0 (default)
 if mod_magnet scripts must process the response body)

Modifying HTTP response status and response body has not changed
and is achieved by setting lua script return value and modifying
the lighty.content lua table.

(note: mod_magnet, mod_setenv, mod_deflate, mod_expire have their
 response start hooks run in the order listed in server.modules)
2020-10-11 12:19:27 -04:00
Glenn Strauss 16a70b9253 [multiple] TLS modules use chunkqueue_peek_data() 2020-10-11 12:19:27 -04:00
Glenn Strauss 6c68e14940 [core] chunkqueue_{peek,read}_data(), squash
chunkqueue_peek_data(), chunkqueue_read_data(), chunkqueue_read_squash()
shared code for chunkqueue manipulation
2020-10-11 12:19:27 -04:00
Glenn Strauss 362be7b3bb [core] relay 1xx from backend over HTTP/1.1
relay 1xx from backend over HTTP/1.1, e.g. 103 Early Hints
(if client is connected using HTTP/1.1)

enabled by default unless disabled in lighttpd.conf with:
  server.feature-flags += ( "server.h1-discard-backend-1xx" = "enable" )

Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response.  Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request.
  https://tools.ietf.org/html/rfc8297#section-3 Security Considerations

x-ref:
  An HTTP Status Code for Indicating Hints (103 Early Hints)
  https://tools.ietf.org/html/rfc8297
2020-10-11 12:19:27 -04:00
Glenn Strauss 869c778aa7 [core] relay 1xx from backend over HTTP/2
relay 1xx from backend over HTTP/2, e.g. 103 Early Hints
(if client is connected using HTTP/2)

enabled by default unless disabled in lighttpd.conf with:
  server.feature-flags += ( "server.h2-discard-backend-1xx" = "enable" )

Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response.  Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request.
  https://tools.ietf.org/html/rfc8297#section-3 Security Considerations

x-ref:
  An HTTP Status Code for Indicating Hints (103 Early Hints)
  https://tools.ietf.org/html/rfc8297
2020-10-11 12:19:26 -04:00
Glenn Strauss 10d9d14633 [core] reload c after chunkqueue_compact_mem()
reload c = cq->first after calling chunkqueue_compact_mem()
2020-10-11 12:19:26 -04:00
Glenn Strauss edbfe36563 [core] support multiple 1xx responses from backend
support multiple 1xx intermediate responses from backends

Currently, all 1xx responses from backends are discarded.
In the future, these 1xx responses may be forwarded to the client
(when lighttpd also configured server.stream-response-body = 1 or = 2)
2020-10-11 12:19:26 -04:00
Glenn Strauss 7a3098398c [core] connection_check_expect_100()
isolate code to check for Expect: 100-continue
(separate out from code to read request body)
2020-10-11 12:19:26 -04:00
Glenn Strauss a330746f06 [TLS] error if inherit empty TLS cfg from globals
error if $SERVER["socket"] inherits empty TLS config from global scope
and ssl.engine = "enable" in the $SERVER["socket"]
2020-10-11 12:19:26 -04:00
Glenn Strauss 5ba175269a [core] skip ls-hpack decode work unused by lighttpd
define LSHPACK_DEC_CALC_HASH 0 to skip work whose results are not used
by lighttpd
2020-10-11 12:19:26 -04:00
Glenn Strauss 3fbb5773e9 [core] http_header_e <=> lshpack_static_hdr_idx
map enum http_header_e to enum lshpack_static_hdr_idx
map enum lshpack_static_hdr_idx to enum http_header_e
2020-10-11 12:19:26 -04:00
Glenn Strauss 367f30a645 [multiple] extend enum http_header_e list 2020-10-11 12:19:26 -04:00
Glenn Strauss 2e0676fd6d [core] extend (data_string *) to store header id
(optional addition to (data_string *), used by http_header.[ch])

extend (data_string *) instead of creating another data_* TYPE_*
  (new data type would probably have (data_string *) as base class)
  (might revisit choice in the future)

HTTP_HEADER_UNSPECIFIED has been removed.  It was used in select
locations as an optimization to avoid looking up enum header_header_e
before checking the array, but the ordering in the array now relies
on having the id.  Having the id allows for a quick check if a common
header is present or not in the htags bitmask, before checking the
array, and allows for integer comparison in the log(n) search of the
array, instead of strncasecmp().

With HTTP_HEADER_UNSPECIFIED removed, add optimization to set bit
in htags for HTTP_HEADER_OTHER when an "other" header is added,
but do not clear the bit, as there might be addtl "other" headers
2020-10-11 12:19:26 -04:00
Glenn Strauss 68ec5ad642 [core] array.[ch] using uint32_t instead of size_t 2020-10-11 12:19:26 -04:00
Glenn Strauss 92e5a021d2 [core] http_header_hkey_get_lc() for HTTP/2
header field-names are required to be lowercase for HTTP/2
so specialize http_header_hkey_get() as http_header_hkey_get_lc()
for simpler comparison

lowercase field-names in http_headers[], as it does not matter for
buffer_eq_icase_ssn(), which is used with http_headers[] for HTTP/1.x
2020-10-11 12:19:26 -04:00
Glenn Strauss babf0d0e4c [core] separate http_header_e from _htags bitmask
separate http_header_e index from r->{rqst,resp}_htags bitmask
(allows http_header_e remain 32-bit and to be used in array indexes
 while also allowing r->{rqst,resp}_htags to grow to 64-bits wide
 in the future, without requiring invasive changes)
2020-10-11 12:19:26 -04:00
Glenn Strauss 9c8981a7d1 [core] tst,set,clr macros for r->{rqst,resp}_htags 2020-10-11 12:19:26 -04:00
Glenn Strauss c58b95f297 [core] light_isupper(), light_islower()
more efficient char checks
(replace one comparision and one branch with one subtraction)
2020-10-11 12:19:26 -04:00
Glenn Strauss 327de98b38 [core] adjust response header prep for common case 2020-10-11 12:19:26 -04:00
Glenn Strauss c88d373507 [core] Content-Length in http_response_send_file() 2020-10-11 12:19:26 -04:00
Glenn Strauss c3a85c9bf5 [mod_wolfssl] standalone module
standalone module forked from mod_openssl
2020-10-11 12:19:26 -04:00
Glenn Strauss 1d9709b8ab [core] update HTTP status codes list from IANA 2020-10-11 12:19:26 -04:00
Glenn Strauss 38620b10bc [core] cold h2_log_response_header*() funcs 2020-10-11 12:19:26 -04:00
Glenn Strauss 97e314fc9e [multiple] inline chunkqueue_length() 2020-10-11 12:19:26 -04:00
Glenn Strauss 3a766d3d02 [mod_webdav] preserve bytes_out when chunks merged 2020-10-11 12:19:26 -04:00
Glenn Strauss 9f8a896891 [mod_ssi] update chunk after shell output redirect
chunkqueue_update_file() to update size of file chunk after
redirecting shell output to file fd
2020-10-11 12:19:26 -04:00
Glenn Strauss dbd458a0e6 [core] set r->state in h2.c
allow h2.c layer to modify r->state in addition to r->h2state

Do not retire/release r on error if trailers (as opposed to headers)
(stream r was just allocated in the same scope for headers,
 so ok to release)
2020-10-11 12:19:26 -04:00
Glenn Strauss 4474e6117f [core] log stream id with debug.log-state-handling
log stream id with debug.log-state-handling in
connection_state_machine_loop()

(id will be 0 for HTTP/2 connection, but not for streams on connection)
(id will be 0 for HTTP/1.x requests)
2020-10-11 12:19:26 -04:00
Glenn Strauss 05ff9c570e [core] combine Cookie request headers with ';'
(thx avij)

clients should send a single Cookie header with multiple cookie values
separated with ';'.
  https://tools.ietf.org/html/rfc6265#section-4.2.1

However, HTTP/2 loosens this requirement for Cookie.
  https://tools.ietf.org/html/rfc7540#section-8.1.2
  Section 8.1.2.5 Compressing the Cookie Header Field
and some HTTP/2 clients (Chrome, Firefox) send multiple
'cookie:' headers in a HEADERS frame.
2020-10-11 12:19:26 -04:00
Glenn Strauss c946b198a3 [core] code size: disable XXH64(), XXH3()
only XXH32() is used by ls-hpack, so disable XXH64() and XXH3()
to reduce code size.  (This maybe be changed in the future if
XXH64() or XXH3() are utilitized for their performance)
2020-10-11 12:19:26 -04:00
Glenn Strauss f52a10545a [mod_openssl] clarify trace w/ deprecated options 2020-10-11 12:19:26 -04:00
Glenn Strauss f2e4e83a85 [mod_dirlisting] fix config parsing crash
(bug on master branch; never released)

(thx avij)

update doc that dir-listing.hide-dotfiles = "enable" by default
since lighttpd 1.4.40 https://redmine.lighttpd.net/issues/1081
2020-10-11 12:19:26 -04:00
Glenn Strauss 32d4569512 [core] xxhash no inline hints; let compiler choose 2020-10-11 12:19:26 -04:00
Glenn Strauss a25aeaa2f6 [build] remove ls-hpack/deps
xxHash is provided separately by lighttpd src/algo_xxhash.[ch]
2020-10-11 12:19:26 -04:00
Glenn Strauss 5ec59999dd [build] modify build, includes for xxHash v0.8.0 2020-10-11 12:19:26 -04:00
Glenn Strauss 85c1a1743e [core] import xxHash v0.8.0
import xxhash.[ch] as algo_xxhash.[ch]

https://github.com/Cyan4973/xxHash
tag: v0.8.0 94e5f23e736f2bb67ebdf90727353e65344f9fc0
2020-10-11 12:19:26 -04:00
Glenn Strauss 156e1da2ae [core] rename splaytree.[ch] to algo_splaytree.[ch] 2020-10-11 12:19:26 -04:00
Glenn Strauss 3ffb195532 [core] move djbhash(), dekhash() to algo_md.h 2020-10-11 12:19:26 -04:00
Glenn Strauss 6c8160e5a4 [core] rename md5.[ch] to algo_md5.[ch] 2020-10-11 12:19:26 -04:00
Glenn Strauss 044c8e80e5 [core] use djbhash in gw_backend to choose host
djbhash() is faster and more cache-friendly than is crc32c()
2020-10-11 12:19:26 -04:00
Glenn Strauss a3af9833c6 [core] fix crash on master if blank line request
(bug on master branch; never released)

(thx avij)

fix crash on master if blank line precedes HTTP/1.1 keep-alive request

header parsing code previously made assumptions that request was
HTTP/1.0 or HTTP/1.1, where a request-line was required, and which
would error out elsewhere if request-line was missing.  The parsing
code also previously looked for "\r\n\r\n" to end headers.

The header offset parsing code was modified and invalidated the above
assumptions, now looking only for blank line "\r\n", but the calling
code had not properly been updated.  (until this patch)
2020-10-11 12:19:26 -04:00
Glenn Strauss 5a694281da [core] ls-hpack optimizations
define LSHPACK_DEC_HTTP1X_OUTPUT 0
  lighttpd does not require HTTP/1.1 output compat from HPACK decoder
  ("field-name: value\r\n")

define NDEBUG (in ls-hpack/lshpack.c)
  lighttpd spends upwards of 20% total lighttpd CPU time in HPACK
  encode/decode in h2load test on static file over cleartext (not TLS)
  Defining NDEBUG eliminates some asserts() and results in a small
  but measurable reduction in CPU usage
2020-10-11 12:19:26 -04:00
Glenn Strauss 4656d91889 [core] document algorithm used in lighttpd etag
Donald E. Knuth
The Art Of Computer Programming Volume 3
Chapter 6.4, Topic: Sorting and Search
2020-10-11 12:19:26 -04:00
Glenn Strauss 5547530a01 [core] do not require '\0' term for k,v hdr parse
no longer require '\0' terminated z-string for key,value header parsing
  http_request_parse_single_header()
  http_header_str_contains_token()
2020-10-11 12:19:26 -04:00
Glenn Strauss 5fd8a26a75 [core] defer optimization to read small files
defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured
2020-10-11 12:19:24 -04:00
Glenn Strauss 3baef447b3 [core] fix crash on master after graceful restart
(bug on master branch; never released)

(thx nate)
2020-10-11 11:43:06 -04:00
Glenn Strauss d8b88be056 [core] move connections_get_state to connections.c
connections_get_state() is used only in connections.c for debugging
2020-10-11 11:43:06 -04:00
Glenn Strauss 0f9ec8b755 [mod_status] dedicated funcs for r->state labels
fork connection_get_state() for use by mod_status, which
might in the future choose to display different labels.
move and rename connection_get_short_state() into mod_status,
as the func is used only by mod_status

(connection_get_state() is currently used elsewhere only for debugging
 in connections.c)
2020-10-11 11:43:06 -04:00
Glenn Strauss 878073d1c9 [core] keep pool of (request_st *) for HTTP/2 2020-10-11 11:43:06 -04:00
Glenn Strauss 550609c8ae [core] request_acquire(), request_release()
continue code abstraction for (request_st *)
2020-10-11 11:43:06 -04:00
Glenn Strauss 0905b6f7f9 [core] fix 1-char trunc of default server.tag
(thx avij)
2020-10-11 11:43:06 -04:00
Glenn Strauss cf0098eac8 [mod_auth] fix crash if auth.require misconfigured (fixes #3023)
(thx veyrdite)

x-ref:
  "Segfault with mod_auth & htpasswd (lighttpd.conf misconfig)"
  https://redmine.lighttpd.net/issues/3023
2020-10-11 11:43:06 -04:00
Glenn Strauss cae205ad99 [build] fix meson build 2020-10-11 11:43:06 -04:00
Glenn Strauss 79e2f5baee [core] ignore empty field-name from backends
Empty field-name is already ignored when generating response headers,
but this is an explicit skip of empty field-name sent from backends,
including field-names that look like HTTP/2 pseudo-headers, as those
begin with ':'.
2020-10-11 11:43:06 -04:00
Glenn Strauss 8e2cf55250 [core] redirect to dir using relative-path
Location response header is permitted to use relative-path in
RFC 7231 Section 7.1.2. Location

Prefer relative path in redirection for the benefit of reverse proxies
and CDNs.  Doing so also avoids potentially disclosing internal schemes
and server names which client might not be able to directly reach.

To restore prior behavior of sending a fully-qualified absolute URI:
server.feature-flags += ("absolute-dir-redirect" => "enable")

x-ref:
  https://bz.apache.org/bugzilla/show_bug.cgi?id=63357
2020-10-11 11:43:06 -04:00
Glenn Strauss baa651a3ab [mod_status] adjust to print HTTP/2 requests 2020-10-11 11:43:06 -04:00
Glenn Strauss 9d74aa600b [mod_status] separate funcs to print request table 2020-10-11 11:43:06 -04:00
Glenn Strauss 06df2716f2 [core] comment possible future ls-hpack optimize 2020-10-11 11:43:06 -04:00
Glenn Strauss 8fc8ab891a [core] http_request_parse_header() specialized
http_request_parse_header() specialized for HTTP/2 request headers
to be parsed as each field-name and value is HPACK-decoded; send headers
directly from HPACK decoder, rather than double-buffering in chunkqueue

http_request_headers_process_h2() for post-processing
2020-10-11 11:43:06 -04:00
Glenn Strauss ada09a23b0 [core] h2_send_headers() specialized for resp hdrs
specialized version of http_response_write_header(); send headers
directly to HPACK encoder, rather than double-buffering in chunkqueue
2020-10-11 11:43:06 -04:00
Glenn Strauss 014e5240ef [core] HTTP/2 HPACK using LiteSpeed ls-hpack
(experimental)
2020-10-11 11:43:06 -04:00
Glenn Strauss 70b1af0640 [core] link in ls-hpack (EXPERIMENTAL)
LiteSpeed ls-hpack v2.2.1

XXX: might be better to include this as a git submodule
but minor code changes were made here for portability:
- C99 flexible array members defined as a[] instead of a[0])
- pedantic compiler warnings (excess ';' and missing declarations)
- deletion of large tables from ls-hpack/huff-tables.h (code size)
2020-10-11 11:43:06 -04:00
Glenn Strauss 1bf4555800 [core] update comment 2020-10-11 11:43:06 -04:00
Glenn Strauss 501182dc4b [core] adjust (disabled) debug code 2020-10-11 11:43:06 -04:00
Glenn Strauss 1c7c19a9d5 [core] skip write throttle for 100 Continue
(small code simplification)
2020-10-11 11:43:06 -04:00
Glenn Strauss 6fc8ad3194 [core] discard 100 102 103 responses from backend 2020-10-11 11:43:06 -04:00
Glenn Strauss 4d9c54f61b [core] mark some error paths with attribute cold 2020-10-11 11:43:06 -04:00
Glenn Strauss 8d7e9439b5 [core] connection transition to HTTP/2; incomplete
(experimental)

(basic framework with some stub functions; incomplete)
(subsequently incrementally updated using git rebase)

HTTP/2 via TLS ALPN extension  (TLS)
HTTP/2 via Upgrade: h2c        (cleartext)
HTTP/2 via Prior Knowledge     (cleartext)

server.feature-flags += ("server.h2proto" => "enable")
to enable HTTP/2
server.feature-flags += ("server.h2c" => "enable")
to enable upgrade to HTTP/2 over cleartext http
2020-10-11 11:43:06 -04:00
Glenn Strauss 784f1ac1da [core] alternate between two joblists
process one joblist and queue work up on second joblist

loop to poll, but poll immediate if work was queued on second joblist
2020-10-11 11:43:06 -04:00
Glenn Strauss 6eca864c5e [core] h2.[ch] with stub funcs (incomplete)
(subsequently incrementally updated using git rebase)

huge props and many thank yous to writers of testing tools used while
developing HTTP/2 support in lighttpd:

  h2spec - conformance testing tool for HTTP/2 implementation
           https://github.com/summerwind/h2spec
  h2load - HTTP/2 benchmarking tool
           https://nghttp2.org/documentation/h2load-howto.html
  curl   - command line tool and library for transferring data with URLs
           https://curl.haxx.se/
2020-10-11 11:43:02 -04:00
Glenn Strauss 2c8f1b4621 [core] isolate more resp code in response.c
http_response_handler() and response generation flow control logic
2020-10-03 09:05:39 -04:00
Glenn Strauss 2f2eec18fb [multiple] isolate more con code in connections.c
move code from connections-glue.c back into connections.c

move code from connections-glue.c to http-header-glue.c
  rename connection_response_reset()
      to http_response_reset()
  rename connection_handle_read_post_error()
      to http_response_reqbody_read_error()
2020-10-03 09:05:38 -04:00
Glenn Strauss 8940fec894 [multiple] modules read reqbody via fn ptr
r->con->reqbody_read() replaces connection_handle_read_post_state()

future: might provide different callbacks for request body with
Content-Length versus request body sent via Transfer-Encoding: chunked
2020-10-03 09:05:38 -04:00
Glenn Strauss 4e4026703d [core] reqpool.[ch] for (request_st *)
move request_init() request_reset() request_free() from connections.c
2020-10-03 09:05:38 -04:00
Glenn Strauss 6971c6c811 [core] move http_request_headers_process()
move http_request_headers_process() to request.[ch]
2020-10-03 09:05:38 -04:00
Glenn Strauss db7b51a4f6 [core] http_header_parse_hoff()
moved and renamed from connections.c:connection_read_header_hoff()
2020-10-03 09:05:38 -04:00
Glenn Strauss 89b3d73daa [core] adjust http_request_headers_process()
adjust for code reuse, and moving routine to lower-level file
2020-10-03 09:05:38 -04:00
Glenn Strauss 517e32785e [core] HTTP2-Settings 2020-10-03 09:05:38 -04:00
Glenn Strauss 4e656f5354 [core] connection_set_fdevent_interest()
extracted from connection_state_machine() (for reuse)

include interest for POLLRDHUP unless already received
2020-10-03 09:05:38 -04:00
Glenn Strauss d4937e29f1 [mod_accesslog,mod_rrdtool] HTTP/2 basic accounting
Note: rrdtool counts do not include HTTP/2 protocol overhead.
Continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
2020-10-03 09:05:38 -04:00
Glenn Strauss afc2025d8e [core] reset connection counters per connection
reset connection counters per connection, not per request

adjust mod_accesslog and mod_rrdtool usage

continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
2020-10-03 09:05:38 -04:00
Glenn Strauss 8eea3bd014 [core] connection_state_machine_loop()
extracted from connection_state_machine() (for reuse)
2020-10-03 09:05:38 -04:00
Glenn Strauss 68dd6312a2 [core] request_headers_process()
extract code from connection_handle_read_state() for future reuse
2020-10-03 09:05:38 -04:00
Glenn Strauss f37c90bccc [core] parse h2 pseudo-headers 2020-10-03 09:05:38 -04:00
Glenn Strauss 4c699be6c6 [core] recognize status: 421 Misdirected Request 2020-10-03 09:05:38 -04:00
Glenn Strauss 77057a7c02 [core] lowercase response hdr field names for h2 2020-10-03 09:05:38 -04:00
Glenn Strauss e089facf84 [core] do not send Connection: close if h2 2020-10-03 09:05:38 -04:00
Glenn Strauss 21987c863f [mod_extforward] preserve changed addr for h2 con
Preserve changed addr for lifetime of h2 connection; upstream proxy
should not reuse same h2 connection for requests from different clients
2020-10-03 09:05:38 -04:00
Glenn Strauss bbcc2f229a [multiple] allow TLS ALPN "h2" if "server.h2proto" 2020-10-03 09:05:38 -04:00
Glenn Strauss a8f8d5edc0 [core] HTTP_VERSION_2 2020-08-13 15:05:25 -04:00
Glenn Strauss 1041a56dd9 [core] server.feature-flags to enable h2
server.feature-flags = ("server.h2proto" => "enable") to enable h2
2020-08-13 15:05:25 -04:00
Glenn Strauss 96324fbba0 [mod_extforward] skip after HANDLER_COMEBACK
do not re-run mod_extforward uri handler after HANDLER_COMEBACK

add some additional comments
2020-08-13 15:05:25 -04:00
Glenn Strauss 0c7d250034 [mod_extforward] save proto per connection
Even though request headers are per-request, update the proto on the
connection level for trusted (proxy) clients.  Note: the proxy must use
each connection only for a single connection from a single client
(typically true in practice), and not for multiple clients.

proto was previously saved on connection level for mod_extforward with
HAProxy PROXY protocol, but did not occur with X-Forwarded-Proto or the
Forwarded request header with proto=...

Before this change, modules which returned HANDLER_COMEBACK could lose
HTTPS=on if the resulting request was for a backend such as mod_fastcgi.
This was reported in mod_rewrite, but could also affect mod_magnet if
MAGNET_RESTART_REQUEST, or mod_cgi with cgi.local-redir = "enable"

x-ref:
  "FastCGI behavior different when using rewrite?"
  https://redmine.lighttpd.net/boards/2/topics/9293
2020-08-13 13:05:12 -04:00
Glenn Strauss b600b75f20 [core] inline buffer_reset()
buffer_reset() is used on pre-allocated buffers, so remove NULL check
2020-08-10 20:05:02 -04:00
Glenn Strauss e1bb579361 [core] disperse settings.h to appropriate headers 2020-08-10 20:04:57 -04:00
Glenn Strauss 3e046ccabf [core] dlsym for FAMNoExists() for compat w/ fam
gamin should be used instead of fam; fam is no longer maintained

This patch makes it safe to build lighttpd with gamin, but run on
systems with the (deprecated) fam installed, which can happen due
to historical package dependency declarations on some platforms
(Debian, Ubuntu).

gamin and fam are not 100% binary compatible.
(Among other things, fam does not provide FAMNoExists())

x-ref:
  "llibgamin vs libfam conflict solving"
  https://salsa.debian.org/debian/lighttpd/-/merge_requests/18
  "libgamin0: libfam shlib dependency wrongly set to libfam0"
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510368
  "undefined symbol: FAMNoExists"
  https://bugs.launchpad.net/bugs/1453463
2020-08-10 15:56:28 -04:00
Glenn Strauss d2d5f27f80 [core] chunkqueue_compact_mem() must upd cq->last 2020-08-10 12:54:33 -04:00
Glenn Strauss 346280fed7 [mod_proxy] do not forward Expect: 100-continue
do not forward Expect: 100-continue to backend
since we do not handle HTTP/1.1 100 Continue response
2020-08-10 12:54:33 -04:00
Glenn Strauss 8b4abaf498 [mod_webdav] fix fallback if linkat() fails
fix fallback if linkat() fails

check at startup if /proc/self/fd is present on systems with O_TMPFILE
(containers might not mount /proc)

x-ref:
  "mod_webdav - PUT files with < 64kb Content-Length reults in zero length file"
  https://redmine.lighttpd.net/boards/2/topics/9273
2020-08-10 12:54:33 -04:00
Glenn Strauss eb4f953396 [build] avoid accept_filter_arg compiler warning
avoid compiler warning for struct accept_filter_arg af_name strncpy()
2020-08-02 12:55:18 -04:00
Glenn Strauss f358f04bdc [build] SCons build mod_deflate w/ libm for brotli 2020-08-02 12:32:37 -04:00
Glenn Strauss 4663564c8a [mod_authn_dbi,mod_authn_mysql] fix coverity issue 2020-08-02 09:00:17 -04:00
Glenn Strauss 7bc7082d32 [mod_extforward] attempt to quiet Coverity warning 2020-08-02 09:00:17 -04:00
Glenn Strauss 4a1960958a [core] add some missing standard includes 2020-08-02 07:47:42 -04:00
Glenn Strauss b2d7635801 [mod_webdav] fix inadvertent string truncation
(bug on master branch; never released)
2020-08-02 07:47:42 -04:00
Glenn Strauss 0fb391c096 [core] http_header_remove_token() 2020-08-02 07:47:42 -04:00
Glenn Strauss d44a26bfa3 [tests] test_base64.c clear buf vs reset
(can be slightly faster to clear buffer than to reset buffer)
2020-08-02 07:47:42 -04:00
Glenn Strauss f7919c1ae3 [mod_proxy] send HTTP/1.1 requests to backends
For prior behavior (HTTP/1.0 requests to backend), force HTTP/1.0 with:
  server.feature-flags = ("proxy.force-http10" => "enable")
2020-08-02 07:47:42 -04:00
Glenn Strauss c3073b4da6 [core] stricter parsing of POST chunked block hdr 2020-08-02 07:47:42 -04:00
Glenn Strauss 9ca668c870 [mod_fastcgi] decode Transfer-Encoding: chunked
decode Transfer-Encoding: chunked from mod_fastcgi backend
2020-08-02 07:47:42 -04:00
Glenn Strauss 7420526ddb [core] decode Transfer-Encoding: chunked from gw
decode Transfer-Encoding: chunked from gw (gateway backends)

Transfer-Encoding: chunked is a hop-by-hop header.

Handling chunked encoding remove a hurdle for mod_proxy to send HTTP/1.1
requests to backends and be able to handle HTTP/1.1 responses.

Other backends ought not to send Transfer-Encoding: chunked, but in
practice, some implementations do.
2020-08-02 07:47:42 -04:00
Glenn Strauss dbd438cca0 [mod_openssl] read secret data w/ BIO_new_mem_buf
Using BIO_new_mem_buf() allows lighttpd to control file reads and
securely wiping memory, avoiding the use of BIO_s_file(), which
employs stdio and BIO internal copying without securely wiping memory.

BIO_new_mem_buf() is also much more performant than BIO_s_file()
or BIO_new_fd() without a buffering BIO, which might make many
syscalls reading a single character at a time.

BIO_new_mem_buf() appears to be supported by older OpenSSL versions
as well as by numerous OpenSSL-compatible APIs, e.g. in BoringSSL,
LibreSSL, WolfSSL
2020-08-02 07:47:42 -04:00
Glenn Strauss 4b755d2b03 [mod_deflate] quicker check for Content-Encoding
The response is not a candidate for mod_deflate if
Content-Encoding was already set by something else.
2020-08-02 07:47:42 -04:00
Glenn Strauss 496dc6d9c5 [core] detect and reject TLS connect to cleartext
detect and reject TLS connection to cleartext listening port
(alternative to failing to receive HTTP header and waiting to time out)
2020-08-02 07:47:42 -04:00
Glenn Strauss 87428818a1 [mod_nss] func renames for consistency
mod_nss was written using mod_mbedtls and mod_gnutls as templates,
but some code was accidentally committed with those other modules'
naming.
2020-08-02 07:47:41 -04:00
Glenn Strauss 33c8cf41db [multiple] rename connection_reset hook to request
rename connection_reset to handle_request_reset
2020-08-02 07:47:41 -04:00
Glenn Strauss bb6a5d7821 [core] separate funcs to reset (request_st *)
request_init(), request_free(), request_reset() separate from (and
called by) connection_init(), connection_free(), connection_reset()
2020-08-02 07:47:41 -04:00
Glenn Strauss 164f7600b7 [multiple] con hooks store ctx in con->plugin_ctx
modules with connection level hooks now store ctx in con->plugin_ctx
2020-08-02 07:47:41 -04:00
Glenn Strauss fea5bdc873 [core] con layer plugin_ctx separate from request 2020-08-02 07:47:41 -04:00
Glenn Strauss 5f7bf472c7 [core] server.feature-flags extensible config
server.feature-flags = ("server.xxxx" => "enable")
2020-08-02 07:46:27 -04:00
Glenn Strauss bc5d4b3492 [mod_compress] remove mod_compress 2020-08-02 06:46:22 -04:00
Glenn Strauss 8dd33a72dd [mod_deflate] mod_deflate subsumes mod_compress
translate config server.modules "mod_compress" to "mod_deflate"
accept compress.* directives, but issue DEPRECATED warning trace

mod_deflate differences from mod_compress:
- mod_compress compress.filetype was exact match; deflate.mimetypes is
  prefix match (behavior change might compress longer mimetype matches,
  which are likely of similar type and compressability)
- mod_compress always sent entire (compressed) file for Range request
  mod_deflate will stream compress range result (not stored in cache)
- mod_compress would short-circuit request with 403 Forbidden error
  if request file did not exist (stat() failed) (This behavior was
  unfriendly to other handlers)
- mod_compress compress.cache-dir layout differs from deflate.cache-dir
  layout; file cache should be cleared (or renamed) when migrating from
  mod_compress to mod_deflate
- mod_deflate does not issue Vary: Accept-Encoding if request does not
  contain Accept-Encoding.  The identity response can be cache by
  proxies and served to clients.  Historically, some proxies disabled
  caching if any Vary: response was seen.  If the Vary header is
  desirable, mod_deflate code which checks for Accept-Encoding and
  compression type can be moved down a few lines to be below the
  setting of the Vary response header.
2020-08-02 06:46:22 -04:00
Glenn Strauss be6da785c5 [mod_deflate] deflate.cache-dir compressed cache
(achieves feature parity with mod_compress)
2020-08-02 06:46:22 -04:00
Glenn Strauss 5b15c1f449 [mod_deflate] propagate errs from internal funcs
also, pass one less arg to internal funcs
2020-08-02 06:46:22 -04:00
Glenn Strauss b2fe37f0a9 [mod_deflate] fix typo in config option
(typo on master branch; never released)
2020-08-02 06:46:22 -04:00
Glenn Strauss 8d5e237c60 [mod_deflate] Brotli support
configuration option: ./configure --with-brotli
2020-08-02 06:46:22 -04:00
Glenn Strauss be8047d82f [build] build fixes for AIX
x-ref:
  "Compile on AIX conflicting types for 'etag_create'"
  https://redmine.lighttpd.net/boards/2/topics/9242
2020-07-21 12:08:27 -04:00
Glenn Strauss 1a516b4c4c [mod_vhostdb_dbi] allow strings and ints in config
e.g. allow "port" value to be a string or integer instead of requiring
     port be a string
2020-07-16 00:30:20 -04:00
Glenn Strauss 3dca923591 [mod_authn_mysql,file] use crypt() to save stack
use crypt() instead of crypt_r() to save stack space,
as struct crypt_data might be very large.

While crypt() is not thread-safe, lighttpd is single-threaded
2020-07-16 00:29:43 -04:00
Glenn Strauss fed2ecae19 [mod_authn_dbi] authn backend employing DBI 2020-07-16 00:29:43 -04:00
Glenn Strauss dba4a328c2 [core] Nettle assert()s if buffer len > digest sz 2020-07-13 17:39:30 -04:00
Glenn Strauss c3dd167dd9 [mod_openssl] fix build warnings on MacOS X
need undef OPENSSL_NO_STDIO for a few funcs which access filesystem
2020-07-13 17:39:30 -04:00
Glenn Strauss 33ae830291 [mod_auth] accept "nonce-secret" & "nonce_secret" 2020-07-13 17:39:30 -04:00
Glenn Strauss 563fe5f013 [mod_authn_ldap,mod_vhostdb_ldap] add timeout opt (#2805)
auth.backend.ldap.timeout = "2000000"    # quoted-string; microseconds
vhostdb.ldap += ("timeout" => "2000000") # quoted-string; microseconds

Default is 2000000 microseconds (2 secs)

These values are converted to struct timeval and passed to
  ldap_set_option(ld, LDAP_OPT_NETWORK_TIMEOUT, );
  ldap_set_option(ld, LDAP_OPT_TIMEOUT, ...);
if those LDAP_OPT_* values are available (both are OpenLDAP-specific).

x-ref:
  "mod_auth caching"
  https://redmine.lighttpd.net/issues/2805
2020-07-13 17:39:30 -04:00
Glenn Strauss e11514b086 [mod_auth,mod_vhostdb] add caching option (fixes #2805)
auth.cache    = ("max-age" => "600")
vhostdb.cache = ("max-age" => "600")

If specified with an empty array, default max-age is 600 secs (10 mins)
auth.cache    = ()
vhostdb.cache = ()

(Note: cache expiration occurs every 8 seconds, so maximum cache time
 might be up to max-age + 8 seconds)

x-ref:
  "mod_auth caching"
  https://redmine.lighttpd.net/issues/2805
2020-07-13 17:39:27 -04:00
Rosen Penev fd2a12d636 [meson] add missing libmaxminddb section
Fixes the following error when building with -Dwith_maxminddb=true:

meson.build:916:1: ERROR: Unknown variable "libmaxminddb".

A full log can be found at meson-logs/meson-log.txt

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-07-11 01:07:19 -04:00
Rosen Penev 04a7d98cb9 [meson] fix libmariadb dependency
libmariadb is what should be used as only the library portion is used.

Fixes compilation under OpenWrt.

Note that mariadb.pc is a superset that links to libmariadb.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-07-11 01:07:16 -04:00
Glenn Strauss 3057b63ea2 [multiple] address coverity warnings 2020-07-10 22:18:26 -04:00
Glenn Strauss 3119d938cb [mod_webdav] build fix: ifdef live_properties 2020-07-10 21:54:27 -04:00
Glenn Strauss 50ddc8108b [mod_webdav] update rc for PROPFIND allprop
(rc currently ignored in case of PROPFIND allprop)
2020-07-10 21:36:32 -04:00
Glenn Strauss 10b7aedcdb [mod_webdav] fix theoretical NULL dereference
fix theoretical NULL dereference identified by Coverity Scan

possible for PROPFIND with specific atypical choices in lighttpd.conf:
- possible for getcontenttype if no content type matches resource
  and no default type configured in lighttpd.conf
- possible for getetag if etag disabled in lighttpd.conf
2020-07-10 21:34:28 -04:00
Glenn Strauss 2781a3be6d [multiple] address coverity warnings 2020-07-10 21:34:28 -04:00
Glenn Strauss ea58cc6f3f [core] store fd in srv->stdin_fd during setup
(pedantic; no impact)

upon error, server will exit, so the impact of momentarily leaking fd
has no impact.  This commit holds the fd in srv->stdin_fd to address
Coverity warning about leaking fd when using server.bind = "/dev/stdin"
2020-07-10 19:30:24 -04:00
Glenn Strauss 6886e78bca [core] remove unused mbedtls_enabled flag
(initially used during development of mod_mbedtls, but no longer used)
2020-07-10 19:28:12 -04:00
Glenn Strauss 2f5fc68b59 [build] fix typo in src/Makefile.am EXTRA_DIST 2020-07-10 19:27:42 -04:00
Glenn Strauss be0d11414e [cmake] update deps for src/t/test_* 2020-07-10 03:20:37 -04:00
Glenn Strauss 52fb78459f [cmake] update deps for src/t/test_* 2020-07-10 03:13:08 -04:00
Glenn Strauss f85d1f9070 [core] splaytree_djbhash() in splaytree.h (reuse)
rename and move hashme() from stat_cache.c to splaytree_djbhash() in
splaytree.h for code reuse
2020-07-10 00:02:48 -04:00
Glenn Strauss 0d4c8edd8f [mod_mysql_vhost] deprecated; use mod_vhostdb_mysql
add warning at server startup when mod_mysql_vhost is loaded

mod_vhostdb_mysql subsumes mod_mysql_vhost.  Individual mod_mysql_vhost
directives map one-to-one to keywords in vhostdb.mysql = (...) directive
2020-07-09 23:45:06 -04:00
Glenn Strauss d334eaf1ee [core] uint32_t is plenty large for path names 2020-07-09 23:45:04 -04:00
Glenn Strauss 28f1867c11 quiet clang analyzer scan-build warnings
(expansion of buffer_string_lenth() inline function and CONST_BUF_LEN()
 macro, which always check for NULL, appears to cause the analyzer to
 believe that a pointer might be NULL in cases where it otherwise can
 not be NULL)

x-ref:
  http://clang-analyzer.llvm.org/faq.html
2020-07-08 22:51:32 -04:00
Glenn Strauss 98a224a4a2 [mod_openssl] prefer some WolfSSL native APIs
Prefer some WolfSSL native APIs when building with WolfSSL.

However, some functionality in WolfSSL is available only through the
WolfSSL compatibility layer for OpenSSL, so the effort to create a
native mod_wolfssl halted here.
2020-07-08 22:51:32 -04:00
Glenn Strauss ed088f5728 [core] error if s->socket_perms chmod() fails 2020-07-08 22:51:32 -04:00
Glenn Strauss 0ad57da55b [mod_openssl,mbedtls,gnutls,nss] fdevent_load_file
employ fdevent_load_file() to load CRL, X509 cert, and private key files
into memory
2020-07-08 22:51:32 -04:00
Glenn Strauss cc04468762 [core] fdevent_load_file() shared code 2020-07-08 22:51:32 -04:00
Glenn Strauss e00deb5578 [mod_nss] NSS option for TLS (fixes #1218)
(experimental)

WARNING: EXPERIMENTAL code sketch; mod_nss is INCOMPLETE and UNTESTED

mod_nss supports most ssl.* config options supported by mod_openssl

x-ref:
  "alternate ssl backend"
  https://redmine.lighttpd.net/issues/1218
2020-07-08 22:51:32 -04:00
Glenn Strauss 8a1ddce0c2 [mod_openssl] default to set MinProtocol TLSv1.2
(behavior change)

TLSv1.0 and TLSv1.1 are deprecated and no longer supported by major
browsers.

MinProtocol is configurable, if required:
e.g. ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1.1")

x-ref:
  https://news.netcraft.com/archives/2020/03/03/browsers-on-track-to-block-850000-tls-1-0-sites.html
2020-07-08 22:51:32 -04:00
Glenn Strauss 8422fa4573 [mod_gnutls] detect certs marked OCSP Must-Staple 2020-07-08 22:51:32 -04:00
Glenn Strauss f56c8e58e4 [mod_openssl] detect certs marked OCSP Must-Staple 2020-07-08 22:51:32 -04:00
Glenn Strauss 1c5def49f6 [mod_webdav] webdav.opts "propfind-depth-infinity"
webdav.opts = ("propfind-depth-infinity" => "enable")
to allow PROPFIND with Depth: infinity   (default "disable")
2020-07-08 22:51:32 -04:00
Glenn Strauss 6bf0b57787 [mod_webdav] unsafe-propfind-follow-symlink option
webdav.opts = ("unsafe-propfind-follow-symlink" => "enable")

This option is unsafe and unsupported.  This option enables non-standard
behavior.  If it works for you, great.  If it does not work for you,
then too bad.

WebDAV resource and collection concepts do not have an equivalence
to unix symlinks.  If "unsafe-propfind-follow-symlink" is "enable",
then lighttpd mod_webdav PROPFIND handling will follow symlinks
if and only if webdav.is_readonly = "enable" is also set.

Allowing symlinks is unsafe in the general case.  Using WebDAV methods
to MOVE a relative symlink does not update the symlink relative target.
LOCK is on the resource (e.g. a symlink), not the target of the symlink.
COPY replaces the resource (e.g. a symlink), not the target of the
symlink.  There are only a few examples of possibly many more reasons
why using symlinks in a WebDAV-writable collection is unsafe.
2020-07-08 22:51:32 -04:00
Glenn Strauss 1a02cd7e33 [mod_webdav] limit webdav_propfind_dir() recursion 2020-07-08 22:51:32 -04:00
Glenn Strauss 965c47c85a [mod_webdav] store webdav.opts as bitflags 2020-07-08 22:51:31 -04:00
Glenn Strauss a479d08b7c [mod_extforward] config warning for module order
expand config warning for module load order to include other TLS modules
2020-07-08 22:51:31 -04:00
Glenn Strauss c0796ee1dd [mod_gnutls] OCSP stapling (#2469) 2020-07-08 22:51:31 -04:00
Glenn Strauss 12e5e745b0 [mod_gnutls] option to override GnuTLS priority
provide option to override GnuTLS priority string
ssl.openssl.ssl-conf-cmd = ("gnutls-override" => "...")
will *entirely* replace the priority string constructed by mod_gnutls.
Admin is responsible to ensure that the override is complete.

debugging:
A non-zero value for debug.log-ssl-noise = x *in the global scope*
will cause mod_gnutls to print the GnuTLS priority string to the
error log at startup.  debug.log-ssl-noise = 0 in $SERVER["socket"]
scopes can then be used to disable runtime ssl log noise, or
debug.log-ssl-noise can be removed from the global scope after debugging
2020-07-08 22:51:31 -04:00
Glenn Strauss 5bbf0d34ca [mod_openssl] compat with BoringSSL 2020-07-08 22:51:31 -04:00
Glenn Strauss b2b6257c7a [mod_openssl] OCSP stapling (fixes #2469)
Define ssl.stapling-file in lighttpd.conf in same scope as ssl.pemfile

x-ref:
  "OCSP Stapling"
  https://redmine.lighttpd.net/issues/2469
2020-07-08 22:51:31 -04:00
Glenn Strauss 60ddaf26c4 [mod_openssl] basic SSL_CONF_cmd for alt TLS libs
provide *basic* (limited) parsing of ssl.openssl.ssl-conf-cmd
for BoringSSL, LibreSSL, WolfSSL
2020-07-08 22:51:31 -04:00
Glenn Strauss 1ba36fda93 [mod_openssl] move SSL_CTX curve conf to new func 2020-07-08 22:51:31 -04:00
Glenn Strauss 3fbedbced2 [mod_openssl] set SSL_OP_PRIORITIZE_CHACHA
set SSL_OP_PRIORITIZE_CHACHA, if available, when
  ssl.honor-cipher-order = "enable"
2020-07-08 22:51:31 -04:00
Glenn Strauss 7d9052c059 [mod_openssl] compat with WolfSSL 2020-07-08 22:51:31 -04:00
Glenn Strauss b4036938de [mod_openssl] compat with LibreSSL 2020-07-08 22:51:31 -04:00
Glenn Strauss c93ecb32a5 [mod_openssl] disable session cache; prefer ticket
reduce memory usage and overhead

many TLSv1.2 implementations support TLSv1.2 session ticket extension
2020-07-08 22:51:31 -04:00
Glenn Strauss 83fb1aed6f [mod_gnutls] ssl.stek-file to specify encrypt key
difference from mod_openssl:

Admin should schedule an independent job to periodically
generate a new STEK before prior STEK lifetime expires.
Only one STEK is active at a time in mod_gnutls.

(more details in prior commit message for mod_openssl)
2020-07-08 22:51:31 -04:00
Glenn Strauss 3e2e8e6d29 [mod_mbedtls] ssl.stek-file to specify encrypt key
difference from mod_openssl:

Admin should schedule an independent job to periodically
generate a new STEK up to 2 times during key lifetime
(mbedtls internals store up to 2 keys)

(more details in prior commit message for mod_openssl)
2020-07-08 22:51:31 -04:00
Glenn Strauss 32a2145f67 [mod_openssl] ssl.stek-file to specify encrypt key
ssl.stek-file to specify session ticket encryption key (STEK)
If ssl.stek-file is specified, it overrides builtin STEK rotation.
STEK file is checked for changes (stat()) once every 64 seconds.

STEK file should be stored in non-persistent storage,
  e.g. /dev/shm/lighttpd/stek-file  (in memory)
with appropriate permissions set to keep stek-file from being
read by other users.  Where possible, systems should also be
configured without swap.

Admin should schedule an independent job to periodically
generate a new STEK up to 3 times during key lifetime
(lighttpd stores up to 3 keys)

format of binary file is:
   4-byte - format version (always 0; for use if format changes)
   4-byte - activation timestamp
   4-byte - expiration timestamp
  16-byte - session ticket key name
  32-byte - session ticket HMAC encrpytion key
  32-byte - session ticket AES encrpytion key

STEK file can be created with a command such as:
  dd if=/dev/random bs=1 count=80 status=none | \
    perl -e 'print pack("iii",0,time()+300,time()+86400),<>' \
    > STEK-file.$$ && mv STEK-file.$$ STEK-file

The above delays activation time by 5 mins (+300 sec) to allow file to
be propagated to other machines.  (admin must handle this independently)
If STEK generation is performed immediately prior to starting lighttpd,
admin should activate keys immediately (without +300).
2020-07-08 22:51:31 -04:00
Glenn Strauss 953fa4ba4d [mod_openssl] rotate session ticket encryption key
server ticket encryption key (STEK) rotation occurs every 28800 seconds
(8 hours) and upon lighttpd server restart.  While lighttpd is running,
(3) encryption keys are preserved, so tickets expire after 1 day.

If using lighttpd with multiple lighttpd workers, then restarting
lighttpd keeps the STEK in sync between lighttpd workers, though
restarting lighttpd three times a day might not be palatable.
Work is in progress to allow admin to supply new encryption keys.
2020-07-08 22:51:31 -04:00
Glenn Strauss 6d62a498a2 [core] more precise check for request stream flags 2020-07-08 22:51:31 -04:00
Glenn Strauss 7b9230ee51 [core] safe memset calls memset() through volatile
safe_memclear() calls memset() through volatile func ptr
2020-07-08 22:51:31 -04:00
Glenn Strauss 6f2dcc58cd [core] SecureZeroMemory() on _WIN32 2020-07-08 22:51:31 -04:00
Glenn Strauss 10c65e88f7 [mod_openssl] update default DH params
update default DH params to use RFC 7919 FFDHE2048 2048-bit group
if ssl.dh-file is not specified
2020-07-08 22:51:31 -04:00
Glenn Strauss 27dbac34a8 [mod_openssl] use openssl feature define for ALPN
use #ifdef TLSEXT_TYPE_application_layer_protocol_negotiation
rather than checking OPENSSL_VERSION_NUMBER >= 0x10002000 (1.0.2)
2020-07-08 22:51:31 -04:00
Glenn Strauss bdb3040f14 [mod_openssl] use SSL_OP_NO_RENEGOTIATION if avail
use [mod_openssl] SSL_OP_NO_RENEGOTIATION if available
(openssl 1.1.0 and later)
2020-07-08 22:51:31 -04:00
Glenn Strauss 3403382261 [mod_openssl] issue warning for deprecated options 2020-07-08 22:51:31 -04:00
Glenn Strauss d605c83a73 [mod_openssl] enable read-ahead, if set, after SNI
note: not disabling if enabled; extra blocks may already have been read
2020-07-08 22:51:31 -04:00
Glenn Strauss 1fc8a3e1f2 [core] sys-crypto-md.h w/ inline message digest fn
sys-crypto-md.h w/ inline message digest functions; shared code
2020-07-08 22:51:31 -04:00
Glenn Strauss c18f442a63 [multiple] add summaries to top of some modules 2020-07-08 22:51:31 -04:00
Glenn Strauss 10dbe38a92 [core] stricter parse of numerical digits
stricter parse of numerical digits for http status code, port num,
and a few other places.  (stricter parse than that of strtol())

content ranges are still parsed more loosely at points of use
2020-07-08 22:51:31 -04:00
Glenn Strauss 6876c16be0 [core] RFC-strict parse of Content-Length
augment simple strtoll() which allowed number to begin with '+'

This is not exploitable for HTTP Request Smuggling since lighttpd
mod_proxy sends "Connection: close" to backends, and other CGI-based
backends reconstitute CONTENT_LENGTH in the environment without '+'.

(thx Amit Klein, Safebreach)
2020-07-08 22:51:31 -04:00
Glenn Strauss acff179322 [mod_openssl] set chains from callback in 1.0.2+ (#2842)
mod_openssl leverages cert callback in openssl 1.0.2 and later
(SSL_CTX_set_cert_cb())

server certificate chain
  is now set in the cert callback

verify_store (for client certificate verification) (ssl.ca-file)
  is now set in the cert callback

more carefully load sensitive files and clear temporary storage
  (with openssl 1.1.1 and later)

x-ref:
  "Lighttpd Returns Wrong Cert In Multi-cert Set-up"
  https://redmine.lighttpd.net/issues/2842
2020-07-08 22:51:31 -04:00
Glenn Strauss 01b1f16b3f [mod_openssl] set cert from callback in 1.0.2+ (fixes #2842)
set server certificate from callback in openssl 1.0.2 and later
(SSL_CTX_set_cert_cb())

For existing versions of lighttpd, certificate selection influenced by
ssl.cipher-list which can be used to set server cipher order preference
(along with ssl.honor-cipher-order = "enable", which is the default)

x-ref:
  "Lighttpd Returns Wrong Cert In Multi-cert Set-up"
  https://redmine.lighttpd.net/issues/2842
  "lighttpd uses wrong pem-file"
  https://redmine.lighttpd.net/issues/3009
2020-07-08 22:51:31 -04:00
Glenn Strauss 2a5b7c648a [mod_openssl] rotate session ticket encryption key
server ticket encryption key (STEK) rotation occurs every 86400 seconds
and upon lighttpd server restart.  If using lighttpd with multiple
lighttpd workers, then restarting lighttpd keeps the STEK in sync
between lighttpd workers.
2020-07-08 22:51:31 -04:00
Glenn Strauss bf4054f8ec [mod_gnutls] GnuTLS option for TLS (fixes #109)
(experimental)

mod_gnutls supports most ssl.* config options supported by mod_openssl

x-ref:
  "GnuTLS support for the mod_ssl"
  https://redmine.lighttpd.net/issues/109
2020-07-08 22:51:31 -04:00
Glenn Strauss 68d626fa9f [mod_openssl] SSL_R_UNEXPECTED_EOF_WHILE_READING
report SSL_R_UNEXPECTED_EOF_WHILE_READING if debug.log-ssl-noise enabled

SSL_R_UNEXPECTED_EOF_WHILE_READING
added in openssl 1.1.1e and reverted in 1.1.1f, but kept in 3.0 branch
2020-07-08 22:51:31 -04:00
Glenn Strauss b33b6f9121 [mod_openssl] do not log ECONNRESET unless debug
do not log ECONNRESET to errorlog unless debug.log-ssl-noise = "enable"
2020-07-08 22:51:31 -04:00
Glenn Strauss 446866333f [multiple] use *(unsigned char *) with ctypes 2020-07-08 22:51:31 -04:00
Glenn Strauss 455dc03778 [core] prefer getxattr() instead of get_attr()
(when lighttpd is built ./configure --with-attr)
2020-07-08 22:51:31 -04:00
Glenn Strauss cb753ec5b5 [mod_mbedtls] mbedTLS option for TLS
(experimental)

mod_mbedtls supports most ssl.* config options supported by mod_openssl

thx Ward Willats for the initial discussion and attempt in the comments
  https://redmine.lighttpd.net/boards/3/topics/7029
2020-07-08 22:51:31 -04:00
Glenn Strauss 7de51cc77b [core] add seed before openssl RAND_pseudo_bytes() 2020-07-08 19:54:30 -04:00
Glenn Strauss ce7840504d [mod_openssl] disable cert vrfy if ALPN acme-tls/1
disable client cert verification if ALPN "acme-tls/1"
2020-07-08 19:54:30 -04:00
Glenn Strauss f7bac374ee [mod_accesslog] process backslash-escapes in fmt
Process basic backslash-escapes in format string from lighttpd.conf
Supported sequences: \a \b \f \n \r \t \v
Other backslash-sequences are replaces with the char following backslash

(Apache mod_log_config supports \n and \t as special-cases)
2020-07-08 19:54:30 -04:00
Glenn Strauss f70912ee44 [mod_webdav] define PATH_MAX if not defined
(PATH_MAX does not appear to be defined on hurd-i386)
2020-07-08 19:54:30 -04:00
Glenn Strauss abe61d0445 [core] return EINVAL if stat_cache_get_entry w/o /
return EINVAL if stat_cache_get_entry() called on path w/o leading '/'
2020-07-08 19:54:30 -04:00
Glenn Strauss e6c47d6ecf [mod_cgi] remove redundant calls to set FD_CLOEXEC 2020-07-08 19:54:30 -04:00
Glenn Strauss 95b2774863 [core] skip excess close() when FD_CLOEXEC defined
do not run through close() loop after fork() and before execve()
when FD_CLOEXEC is defined
2020-07-08 19:54:30 -04:00
Glenn Strauss b28a3714c4 [multiple] ./configure --with-nettle to use Nettle
./configure --with-nettle to use Nettle crypto lib for algorithms,
instead of OpenSSL or wolfSSL.  Note: Nettle does not provide TLS.

x-ref:
  "How to use SHA-256 without OpenSSL?"
  https://redmine.lighttpd.net/boards/2/topics/8903
2020-07-08 19:54:30 -04:00
Glenn Strauss 0acb0f3604 [multiple] fix some cc warnings in 32-bit, powerpc 2020-07-08 19:54:30 -04:00
Glenn Strauss c752d4696e [multiple] correct misspellings in comments
x-ref:
  "Script for fixing spelling errors with codespell"
  https://redmine.lighttpd.net/boards/3/topics/8947
2020-07-08 19:54:30 -04:00
Glenn Strauss bcddbe186f [mod_proxy] stream request using HTTP/1.1 chunked (fixes #3006)
stream request body using HTTP/1.1 Transfer-Encoding: chunked

(Note: if backend proxy target does not support HTTP/1.1,
 then do not use server.stream-request-body = 1 or 2)

If not streaming to backend, collect request body
(now supporting Transfer-Encoding: chunked from client
 and then sending with Content-Length to backend)

x-ref:
  "Lighty returns HTTP 411 Length Required with proxy and streaming requests/reponses body"
  https://redmine.lighttpd.net/issues/3006
2020-07-08 19:54:29 -04:00