Commit Graph

2935 Commits

Author SHA1 Message Date
Glenn Strauss 550609c8ae [core] request_acquire(), request_release()
continue code abstraction for (request_st *)
2020-10-11 11:43:06 -04:00
Glenn Strauss 0905b6f7f9 [core] fix 1-char trunc of default server.tag
(thx avij)
2020-10-11 11:43:06 -04:00
Glenn Strauss cf0098eac8 [mod_auth] fix crash if auth.require misconfigured (fixes #3023)
(thx veyrdite)

  "Segfault with mod_auth & htpasswd (lighttpd.conf misconfig)"
2020-10-11 11:43:06 -04:00
Glenn Strauss cae205ad99 [build] fix meson build 2020-10-11 11:43:06 -04:00
Glenn Strauss 79e2f5baee [core] ignore empty field-name from backends
Empty field-name is already ignored when generating response headers,
but this is an explicit skip of empty field-name sent from backends,
including field-names that look like HTTP/2 pseudo-headers, as those
begin with ':'.
2020-10-11 11:43:06 -04:00
Glenn Strauss 8e2cf55250 [core] redirect to dir using relative-path
Location response header is permitted to use relative-path in
RFC 7231 Section 7.1.2. Location

Prefer relative path in redirection for the benefit of reverse proxies
and CDNs.  Doing so also avoids potentially disclosing internal schemes
and server names which client might not be able to directly reach.

To restore prior behavior of sending a fully-qualified absolute URI:
server.feature-flags += ("absolute-dir-redirect" => "enable")

2020-10-11 11:43:06 -04:00
Glenn Strauss baa651a3ab [mod_status] adjust to print HTTP/2 requests 2020-10-11 11:43:06 -04:00
Glenn Strauss 9d74aa600b [mod_status] separate funcs to print request table 2020-10-11 11:43:06 -04:00
Glenn Strauss 06df2716f2 [core] comment possible future ls-hpack optimize 2020-10-11 11:43:06 -04:00
Glenn Strauss 8fc8ab891a [core] http_request_parse_header() specialized
http_request_parse_header() specialized for HTTP/2 request headers
to be parsed as each field-name and value is HPACK-decoded; send headers
directly from HPACK decoder, rather than double-buffering in chunkqueue

http_request_headers_process_h2() for post-processing
2020-10-11 11:43:06 -04:00
Glenn Strauss ada09a23b0 [core] h2_send_headers() specialized for resp hdrs
specialized version of http_response_write_header(); send headers
directly to HPACK encoder, rather than double-buffering in chunkqueue
2020-10-11 11:43:06 -04:00
Glenn Strauss 014e5240ef [core] HTTP/2 HPACK using LiteSpeed ls-hpack
2020-10-11 11:43:06 -04:00
Glenn Strauss 70b1af0640 [core] link in ls-hpack (EXPERIMENTAL)
LiteSpeed ls-hpack v2.2.1

XXX: might be better to include this as a git submodule
but minor code changes were made here for portability:
- C99 flexible array members defined as a[] instead of a[0])
- pedantic compiler warnings (excess ';' and missing declarations)
- deletion of large tables from ls-hpack/huff-tables.h (code size)
2020-10-11 11:43:06 -04:00
Glenn Strauss 1bf4555800 [core] update comment 2020-10-11 11:43:06 -04:00
Glenn Strauss 501182dc4b [core] adjust (disabled) debug code 2020-10-11 11:43:06 -04:00
Glenn Strauss 1c7c19a9d5 [core] skip write throttle for 100 Continue
(small code simplification)
2020-10-11 11:43:06 -04:00
Glenn Strauss 6fc8ad3194 [core] discard 100 102 103 responses from backend 2020-10-11 11:43:06 -04:00
Glenn Strauss 4d9c54f61b [core] mark some error paths with attribute cold 2020-10-11 11:43:06 -04:00
Glenn Strauss 8d7e9439b5 [core] connection transition to HTTP/2; incomplete

(basic framework with some stub functions; incomplete)
(subsequently incrementally updated using git rebase)

HTTP/2 via TLS ALPN extension  (TLS)
HTTP/2 via Upgrade: h2c        (cleartext)
HTTP/2 via Prior Knowledge     (cleartext)

server.feature-flags += ("server.h2proto" => "enable")
to enable HTTP/2
server.feature-flags += ("server.h2c" => "enable")
to enable upgrade to HTTP/2 over cleartext http
2020-10-11 11:43:06 -04:00
Glenn Strauss 784f1ac1da [core] alternate between two joblists
process one joblist and queue work up on second joblist

loop to poll, but poll immediate if work was queued on second joblist
2020-10-11 11:43:06 -04:00
Glenn Strauss 6eca864c5e [core] h2.[ch] with stub funcs (incomplete)
(subsequently incrementally updated using git rebase)

huge props and many thank yous to writers of testing tools used while
developing HTTP/2 support in lighttpd:

  h2spec - conformance testing tool for HTTP/2 implementation
  h2load - HTTP/2 benchmarking tool
  curl   - command line tool and library for transferring data with URLs
2020-10-11 11:43:02 -04:00
Glenn Strauss 2c8f1b4621 [core] isolate more resp code in response.c
http_response_handler() and response generation flow control logic
2020-10-03 09:05:39 -04:00
Glenn Strauss 2f2eec18fb [multiple] isolate more con code in connections.c
move code from connections-glue.c back into connections.c

move code from connections-glue.c to http-header-glue.c
  rename connection_response_reset()
      to http_response_reset()
  rename connection_handle_read_post_error()
      to http_response_reqbody_read_error()
2020-10-03 09:05:38 -04:00
Glenn Strauss 8940fec894 [multiple] modules read reqbody via fn ptr
r->con->reqbody_read() replaces connection_handle_read_post_state()

future: might provide different callbacks for request body with
Content-Length versus request body sent via Transfer-Encoding: chunked
2020-10-03 09:05:38 -04:00
Glenn Strauss 4e4026703d [core] reqpool.[ch] for (request_st *)
move request_init() request_reset() request_free() from connections.c
2020-10-03 09:05:38 -04:00
Glenn Strauss 6971c6c811 [core] move http_request_headers_process()
move http_request_headers_process() to request.[ch]
2020-10-03 09:05:38 -04:00
Glenn Strauss db7b51a4f6 [core] http_header_parse_hoff()
moved and renamed from connections.c:connection_read_header_hoff()
2020-10-03 09:05:38 -04:00
Glenn Strauss 89b3d73daa [core] adjust http_request_headers_process()
adjust for code reuse, and moving routine to lower-level file
2020-10-03 09:05:38 -04:00
Glenn Strauss 517e32785e [core] HTTP2-Settings 2020-10-03 09:05:38 -04:00
Glenn Strauss 4e656f5354 [core] connection_set_fdevent_interest()
extracted from connection_state_machine() (for reuse)

include interest for POLLRDHUP unless already received
2020-10-03 09:05:38 -04:00
Glenn Strauss d4937e29f1 [mod_accesslog,mod_rrdtool] HTTP/2 basic accounting
Note: rrdtool counts do not include HTTP/2 protocol overhead.
Continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
2020-10-03 09:05:38 -04:00
Glenn Strauss afc2025d8e [core] reset connection counters per connection
reset connection counters per connection, not per request

adjust mod_accesslog and mod_rrdtool usage

continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
2020-10-03 09:05:38 -04:00
Glenn Strauss 8eea3bd014 [core] connection_state_machine_loop()
extracted from connection_state_machine() (for reuse)
2020-10-03 09:05:38 -04:00
Glenn Strauss 68dd6312a2 [core] request_headers_process()
extract code from connection_handle_read_state() for future reuse
2020-10-03 09:05:38 -04:00
Glenn Strauss f37c90bccc [core] parse h2 pseudo-headers 2020-10-03 09:05:38 -04:00
Glenn Strauss 4c699be6c6 [core] recognize status: 421 Misdirected Request 2020-10-03 09:05:38 -04:00
Glenn Strauss 77057a7c02 [core] lowercase response hdr field names for h2 2020-10-03 09:05:38 -04:00
Glenn Strauss e089facf84 [core] do not send Connection: close if h2 2020-10-03 09:05:38 -04:00
Glenn Strauss 21987c863f [mod_extforward] preserve changed addr for h2 con
Preserve changed addr for lifetime of h2 connection; upstream proxy
should not reuse same h2 connection for requests from different clients
2020-10-03 09:05:38 -04:00
Glenn Strauss bbcc2f229a [multiple] allow TLS ALPN "h2" if "server.h2proto" 2020-10-03 09:05:38 -04:00
Glenn Strauss a8f8d5edc0 [core] HTTP_VERSION_2 2020-08-13 15:05:25 -04:00
Glenn Strauss 1041a56dd9 [core] server.feature-flags to enable h2
server.feature-flags = ("server.h2proto" => "enable") to enable h2
2020-08-13 15:05:25 -04:00
Glenn Strauss 96324fbba0 [mod_extforward] skip after HANDLER_COMEBACK
do not re-run mod_extforward uri handler after HANDLER_COMEBACK

add some additional comments
2020-08-13 15:05:25 -04:00
Glenn Strauss 0c7d250034 [mod_extforward] save proto per connection
Even though request headers are per-request, update the proto on the
connection level for trusted (proxy) clients.  Note: the proxy must use
each connection only for a single connection from a single client
(typically true in practice), and not for multiple clients.

proto was previously saved on connection level for mod_extforward with
HAProxy PROXY protocol, but did not occur with X-Forwarded-Proto or the
Forwarded request header with proto=...

Before this change, modules which returned HANDLER_COMEBACK could lose
HTTPS=on if the resulting request was for a backend such as mod_fastcgi.
This was reported in mod_rewrite, but could also affect mod_magnet if
MAGNET_RESTART_REQUEST, or mod_cgi with cgi.local-redir = "enable"

  "FastCGI behavior different when using rewrite?"
2020-08-13 13:05:12 -04:00
Glenn Strauss b600b75f20 [core] inline buffer_reset()
buffer_reset() is used on pre-allocated buffers, so remove NULL check
2020-08-10 20:05:02 -04:00
Glenn Strauss e1bb579361 [core] disperse settings.h to appropriate headers 2020-08-10 20:04:57 -04:00
Glenn Strauss 3e046ccabf [core] dlsym for FAMNoExists() for compat w/ fam
gamin should be used instead of fam; fam is no longer maintained

This patch makes it safe to build lighttpd with gamin, but run on
systems with the (deprecated) fam installed, which can happen due
to historical package dependency declarations on some platforms
(Debian, Ubuntu).

gamin and fam are not 100% binary compatible.
(Among other things, fam does not provide FAMNoExists())

  "llibgamin vs libfam conflict solving"
  "libgamin0: libfam shlib dependency wrongly set to libfam0"
  "undefined symbol: FAMNoExists"
2020-08-10 15:56:28 -04:00
Glenn Strauss d2d5f27f80 [core] chunkqueue_compact_mem() must upd cq->last 2020-08-10 12:54:33 -04:00
Glenn Strauss 346280fed7 [mod_proxy] do not forward Expect: 100-continue
do not forward Expect: 100-continue to backend
since we do not handle HTTP/1.1 100 Continue response
2020-08-10 12:54:33 -04:00
Glenn Strauss 8b4abaf498 [mod_webdav] fix fallback if linkat() fails
fix fallback if linkat() fails

check at startup if /proc/self/fd is present on systems with O_TMPFILE
(containers might not mount /proc)

  "mod_webdav - PUT files with < 64kb Content-Length reults in zero length file"
2020-08-10 12:54:33 -04:00
Glenn Strauss eb4f953396 [build] avoid accept_filter_arg compiler warning
avoid compiler warning for struct accept_filter_arg af_name strncpy()
2020-08-02 12:55:18 -04:00
Glenn Strauss f358f04bdc [build] SCons build mod_deflate w/ libm for brotli 2020-08-02 12:32:37 -04:00
Glenn Strauss 4663564c8a [mod_authn_dbi,mod_authn_mysql] fix coverity issue 2020-08-02 09:00:17 -04:00
Glenn Strauss 7bc7082d32 [mod_extforward] attempt to quiet Coverity warning 2020-08-02 09:00:17 -04:00
Glenn Strauss 4a1960958a [core] add some missing standard includes 2020-08-02 07:47:42 -04:00
Glenn Strauss b2d7635801 [mod_webdav] fix inadvertent string truncation
(bug on master branch; never released)
2020-08-02 07:47:42 -04:00
Glenn Strauss 0fb391c096 [core] http_header_remove_token() 2020-08-02 07:47:42 -04:00
Glenn Strauss d44a26bfa3 [tests] test_base64.c clear buf vs reset
(can be slightly faster to clear buffer than to reset buffer)
2020-08-02 07:47:42 -04:00
Glenn Strauss f7919c1ae3 [mod_proxy] send HTTP/1.1 requests to backends
For prior behavior (HTTP/1.0 requests to backend), force HTTP/1.0 with:
  server.feature-flags = ("proxy.force-http10" => "enable")
2020-08-02 07:47:42 -04:00
Glenn Strauss c3073b4da6 [core] stricter parsing of POST chunked block hdr 2020-08-02 07:47:42 -04:00
Glenn Strauss 9ca668c870 [mod_fastcgi] decode Transfer-Encoding: chunked
decode Transfer-Encoding: chunked from mod_fastcgi backend
2020-08-02 07:47:42 -04:00
Glenn Strauss 7420526ddb [core] decode Transfer-Encoding: chunked from gw
decode Transfer-Encoding: chunked from gw (gateway backends)

Transfer-Encoding: chunked is a hop-by-hop header.

Handling chunked encoding remove a hurdle for mod_proxy to send HTTP/1.1
requests to backends and be able to handle HTTP/1.1 responses.

Other backends ought not to send Transfer-Encoding: chunked, but in
practice, some implementations do.
2020-08-02 07:47:42 -04:00
Glenn Strauss dbd438cca0 [mod_openssl] read secret data w/ BIO_new_mem_buf
Using BIO_new_mem_buf() allows lighttpd to control file reads and
securely wiping memory, avoiding the use of BIO_s_file(), which
employs stdio and BIO internal copying without securely wiping memory.

BIO_new_mem_buf() is also much more performant than BIO_s_file()
or BIO_new_fd() without a buffering BIO, which might make many
syscalls reading a single character at a time.

BIO_new_mem_buf() appears to be supported by older OpenSSL versions
as well as by numerous OpenSSL-compatible APIs, e.g. in BoringSSL,
LibreSSL, WolfSSL
2020-08-02 07:47:42 -04:00
Glenn Strauss 4b755d2b03 [mod_deflate] quicker check for Content-Encoding
The response is not a candidate for mod_deflate if
Content-Encoding was already set by something else.
2020-08-02 07:47:42 -04:00
Glenn Strauss 496dc6d9c5 [core] detect and reject TLS connect to cleartext
detect and reject TLS connection to cleartext listening port
(alternative to failing to receive HTTP header and waiting to time out)
2020-08-02 07:47:42 -04:00
Glenn Strauss 87428818a1 [mod_nss] func renames for consistency
mod_nss was written using mod_mbedtls and mod_gnutls as templates,
but some code was accidentally committed with those other modules'
2020-08-02 07:47:41 -04:00
Glenn Strauss 33c8cf41db [multiple] rename connection_reset hook to request
rename connection_reset to handle_request_reset
2020-08-02 07:47:41 -04:00
Glenn Strauss bb6a5d7821 [core] separate funcs to reset (request_st *)
request_init(), request_free(), request_reset() separate from (and
called by) connection_init(), connection_free(), connection_reset()
2020-08-02 07:47:41 -04:00
Glenn Strauss 164f7600b7 [multiple] con hooks store ctx in con->plugin_ctx
modules with connection level hooks now store ctx in con->plugin_ctx
2020-08-02 07:47:41 -04:00
Glenn Strauss fea5bdc873 [core] con layer plugin_ctx separate from request 2020-08-02 07:47:41 -04:00
Glenn Strauss 5f7bf472c7 [core] server.feature-flags extensible config
server.feature-flags = ("server.xxxx" => "enable")
2020-08-02 07:46:27 -04:00
Glenn Strauss bc5d4b3492 [mod_compress] remove mod_compress 2020-08-02 06:46:22 -04:00
Glenn Strauss 8dd33a72dd [mod_deflate] mod_deflate subsumes mod_compress
translate config server.modules "mod_compress" to "mod_deflate"
accept compress.* directives, but issue DEPRECATED warning trace

mod_deflate differences from mod_compress:
- mod_compress compress.filetype was exact match; deflate.mimetypes is
  prefix match (behavior change might compress longer mimetype matches,
  which are likely of similar type and compressability)
- mod_compress always sent entire (compressed) file for Range request
  mod_deflate will stream compress range result (not stored in cache)
- mod_compress would short-circuit request with 403 Forbidden error
  if request file did not exist (stat() failed) (This behavior was
  unfriendly to other handlers)
- mod_compress compress.cache-dir layout differs from deflate.cache-dir
  layout; file cache should be cleared (or renamed) when migrating from
  mod_compress to mod_deflate
- mod_deflate does not issue Vary: Accept-Encoding if request does not
  contain Accept-Encoding.  The identity response can be cache by
  proxies and served to clients.  Historically, some proxies disabled
  caching if any Vary: response was seen.  If the Vary header is
  desirable, mod_deflate code which checks for Accept-Encoding and
  compression type can be moved down a few lines to be below the
  setting of the Vary response header.
2020-08-02 06:46:22 -04:00
Glenn Strauss be6da785c5 [mod_deflate] deflate.cache-dir compressed cache
(achieves feature parity with mod_compress)
2020-08-02 06:46:22 -04:00
Glenn Strauss 5b15c1f449 [mod_deflate] propagate errs from internal funcs
also, pass one less arg to internal funcs
2020-08-02 06:46:22 -04:00
Glenn Strauss b2fe37f0a9 [mod_deflate] fix typo in config option
(typo on master branch; never released)
2020-08-02 06:46:22 -04:00
Glenn Strauss 8d5e237c60 [mod_deflate] Brotli support
configuration option: ./configure --with-brotli
2020-08-02 06:46:22 -04:00
Glenn Strauss be8047d82f [build] build fixes for AIX
  "Compile on AIX conflicting types for 'etag_create'"
2020-07-21 12:08:27 -04:00
Glenn Strauss 1a516b4c4c [mod_vhostdb_dbi] allow strings and ints in config
e.g. allow "port" value to be a string or integer instead of requiring
     port be a string
2020-07-16 00:30:20 -04:00
Glenn Strauss 3dca923591 [mod_authn_mysql,file] use crypt() to save stack
use crypt() instead of crypt_r() to save stack space,
as struct crypt_data might be very large.

While crypt() is not thread-safe, lighttpd is single-threaded
2020-07-16 00:29:43 -04:00
Glenn Strauss fed2ecae19 [mod_authn_dbi] authn backend employing DBI 2020-07-16 00:29:43 -04:00
Glenn Strauss dba4a328c2 [core] Nettle assert()s if buffer len > digest sz 2020-07-13 17:39:30 -04:00
Glenn Strauss c3dd167dd9 [mod_openssl] fix build warnings on MacOS X
need undef OPENSSL_NO_STDIO for a few funcs which access filesystem
2020-07-13 17:39:30 -04:00
Glenn Strauss 33ae830291 [mod_auth] accept "nonce-secret" & "nonce_secret" 2020-07-13 17:39:30 -04:00
Glenn Strauss 563fe5f013 [mod_authn_ldap,mod_vhostdb_ldap] add timeout opt (#2805)
auth.backend.ldap.timeout = "2000000"    # quoted-string; microseconds
vhostdb.ldap += ("timeout" => "2000000") # quoted-string; microseconds

Default is 2000000 microseconds (2 secs)

These values are converted to struct timeval and passed to
  ldap_set_option(ld, LDAP_OPT_NETWORK_TIMEOUT, );
  ldap_set_option(ld, LDAP_OPT_TIMEOUT, ...);
if those LDAP_OPT_* values are available (both are OpenLDAP-specific).

  "mod_auth caching"
2020-07-13 17:39:30 -04:00
Glenn Strauss e11514b086 [mod_auth,mod_vhostdb] add caching option (fixes #2805)
auth.cache    = ("max-age" => "600")
vhostdb.cache = ("max-age" => "600")

If specified with an empty array, default max-age is 600 secs (10 mins)
auth.cache    = ()
vhostdb.cache = ()

(Note: cache expiration occurs every 8 seconds, so maximum cache time
 might be up to max-age + 8 seconds)

  "mod_auth caching"
2020-07-13 17:39:27 -04:00
Rosen Penev fd2a12d636 [meson] add missing libmaxminddb section
Fixes the following error when building with -Dwith_maxminddb=true: ERROR: Unknown variable "libmaxminddb".

A full log can be found at meson-logs/meson-log.txt

Signed-off-by: Rosen Penev <>
2020-07-11 01:07:19 -04:00
Rosen Penev 04a7d98cb9 [meson] fix libmariadb dependency
libmariadb is what should be used as only the library portion is used.

Fixes compilation under OpenWrt.

Note that mariadb.pc is a superset that links to libmariadb.

Signed-off-by: Rosen Penev <>
2020-07-11 01:07:16 -04:00
Glenn Strauss 3057b63ea2 [multiple] address coverity warnings 2020-07-10 22:18:26 -04:00
Glenn Strauss 3119d938cb [mod_webdav] build fix: ifdef live_properties 2020-07-10 21:54:27 -04:00
Glenn Strauss 50ddc8108b [mod_webdav] update rc for PROPFIND allprop
(rc currently ignored in case of PROPFIND allprop)
2020-07-10 21:36:32 -04:00
Glenn Strauss 10b7aedcdb [mod_webdav] fix theoretical NULL dereference
fix theoretical NULL dereference identified by Coverity Scan

possible for PROPFIND with specific atypical choices in lighttpd.conf:
- possible for getcontenttype if no content type matches resource
  and no default type configured in lighttpd.conf
- possible for getetag if etag disabled in lighttpd.conf
2020-07-10 21:34:28 -04:00
Glenn Strauss 2781a3be6d [multiple] address coverity warnings 2020-07-10 21:34:28 -04:00
Glenn Strauss ea58cc6f3f [core] store fd in srv->stdin_fd during setup
(pedantic; no impact)

upon error, server will exit, so the impact of momentarily leaking fd
has no impact.  This commit holds the fd in srv->stdin_fd to address
Coverity warning about leaking fd when using server.bind = "/dev/stdin"
2020-07-10 19:30:24 -04:00
Glenn Strauss 6886e78bca [core] remove unused mbedtls_enabled flag
(initially used during development of mod_mbedtls, but no longer used)
2020-07-10 19:28:12 -04:00
Glenn Strauss 2f5fc68b59 [build] fix typo in src/ EXTRA_DIST 2020-07-10 19:27:42 -04:00
Glenn Strauss be0d11414e [cmake] update deps for src/t/test_* 2020-07-10 03:20:37 -04:00
Glenn Strauss 52fb78459f [cmake] update deps for src/t/test_* 2020-07-10 03:13:08 -04:00
Glenn Strauss f85d1f9070 [core] splaytree_djbhash() in splaytree.h (reuse)
rename and move hashme() from stat_cache.c to splaytree_djbhash() in
splaytree.h for code reuse
2020-07-10 00:02:48 -04:00
Glenn Strauss 0d4c8edd8f [mod_mysql_vhost] deprecated; use mod_vhostdb_mysql
add warning at server startup when mod_mysql_vhost is loaded

mod_vhostdb_mysql subsumes mod_mysql_vhost.  Individual mod_mysql_vhost
directives map one-to-one to keywords in vhostdb.mysql = (...) directive
2020-07-09 23:45:06 -04:00
Glenn Strauss d334eaf1ee [core] uint32_t is plenty large for path names 2020-07-09 23:45:04 -04:00
Glenn Strauss 28f1867c11 quiet clang analyzer scan-build warnings
(expansion of buffer_string_lenth() inline function and CONST_BUF_LEN()
 macro, which always check for NULL, appears to cause the analyzer to
 believe that a pointer might be NULL in cases where it otherwise can
 not be NULL)

2020-07-08 22:51:32 -04:00
Glenn Strauss 98a224a4a2 [mod_openssl] prefer some WolfSSL native APIs
Prefer some WolfSSL native APIs when building with WolfSSL.

However, some functionality in WolfSSL is available only through the
WolfSSL compatibility layer for OpenSSL, so the effort to create a
native mod_wolfssl halted here.
2020-07-08 22:51:32 -04:00
Glenn Strauss ed088f5728 [core] error if s->socket_perms chmod() fails 2020-07-08 22:51:32 -04:00
Glenn Strauss 0ad57da55b [mod_openssl,mbedtls,gnutls,nss] fdevent_load_file
employ fdevent_load_file() to load CRL, X509 cert, and private key files
into memory
2020-07-08 22:51:32 -04:00
Glenn Strauss cc04468762 [core] fdevent_load_file() shared code 2020-07-08 22:51:32 -04:00
Glenn Strauss e00deb5578 [mod_nss] NSS option for TLS (fixes #1218)


mod_nss supports most ssl.* config options supported by mod_openssl

  "alternate ssl backend"
2020-07-08 22:51:32 -04:00
Glenn Strauss 8a1ddce0c2 [mod_openssl] default to set MinProtocol TLSv1.2
(behavior change)

TLSv1.0 and TLSv1.1 are deprecated and no longer supported by major

MinProtocol is configurable, if required:
e.g. ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1.1")

2020-07-08 22:51:32 -04:00
Glenn Strauss 8422fa4573 [mod_gnutls] detect certs marked OCSP Must-Staple 2020-07-08 22:51:32 -04:00
Glenn Strauss f56c8e58e4 [mod_openssl] detect certs marked OCSP Must-Staple 2020-07-08 22:51:32 -04:00
Glenn Strauss 1c5def49f6 [mod_webdav] webdav.opts "propfind-depth-infinity"
webdav.opts = ("propfind-depth-infinity" => "enable")
to allow PROPFIND with Depth: infinity   (default "disable")
2020-07-08 22:51:32 -04:00
Glenn Strauss 6bf0b57787 [mod_webdav] unsafe-propfind-follow-symlink option
webdav.opts = ("unsafe-propfind-follow-symlink" => "enable")

This option is unsafe and unsupported.  This option enables non-standard
behavior.  If it works for you, great.  If it does not work for you,
then too bad.

WebDAV resource and collection concepts do not have an equivalence
to unix symlinks.  If "unsafe-propfind-follow-symlink" is "enable",
then lighttpd mod_webdav PROPFIND handling will follow symlinks
if and only if webdav.is_readonly = "enable" is also set.

Allowing symlinks is unsafe in the general case.  Using WebDAV methods
to MOVE a relative symlink does not update the symlink relative target.
LOCK is on the resource (e.g. a symlink), not the target of the symlink.
COPY replaces the resource (e.g. a symlink), not the target of the
symlink.  There are only a few examples of possibly many more reasons
why using symlinks in a WebDAV-writable collection is unsafe.
2020-07-08 22:51:32 -04:00
Glenn Strauss 1a02cd7e33 [mod_webdav] limit webdav_propfind_dir() recursion 2020-07-08 22:51:32 -04:00
Glenn Strauss 965c47c85a [mod_webdav] store webdav.opts as bitflags 2020-07-08 22:51:31 -04:00
Glenn Strauss a479d08b7c [mod_extforward] config warning for module order
expand config warning for module load order to include other TLS modules
2020-07-08 22:51:31 -04:00
Glenn Strauss c0796ee1dd [mod_gnutls] OCSP stapling (#2469) 2020-07-08 22:51:31 -04:00
Glenn Strauss 12e5e745b0 [mod_gnutls] option to override GnuTLS priority
provide option to override GnuTLS priority string
ssl.openssl.ssl-conf-cmd = ("gnutls-override" => "...")
will *entirely* replace the priority string constructed by mod_gnutls.
Admin is responsible to ensure that the override is complete.

A non-zero value for debug.log-ssl-noise = x *in the global scope*
will cause mod_gnutls to print the GnuTLS priority string to the
error log at startup.  debug.log-ssl-noise = 0 in $SERVER["socket"]
scopes can then be used to disable runtime ssl log noise, or
debug.log-ssl-noise can be removed from the global scope after debugging
2020-07-08 22:51:31 -04:00
Glenn Strauss 5bbf0d34ca [mod_openssl] compat with BoringSSL 2020-07-08 22:51:31 -04:00
Glenn Strauss b2b6257c7a [mod_openssl] OCSP stapling (fixes #2469)
Define ssl.stapling-file in lighttpd.conf in same scope as ssl.pemfile

  "OCSP Stapling"
2020-07-08 22:51:31 -04:00
Glenn Strauss 60ddaf26c4 [mod_openssl] basic SSL_CONF_cmd for alt TLS libs
provide *basic* (limited) parsing of ssl.openssl.ssl-conf-cmd
for BoringSSL, LibreSSL, WolfSSL
2020-07-08 22:51:31 -04:00
Glenn Strauss 1ba36fda93 [mod_openssl] move SSL_CTX curve conf to new func 2020-07-08 22:51:31 -04:00
Glenn Strauss 3fbedbced2 [mod_openssl] set SSL_OP_PRIORITIZE_CHACHA
set SSL_OP_PRIORITIZE_CHACHA, if available, when
  ssl.honor-cipher-order = "enable"
2020-07-08 22:51:31 -04:00
Glenn Strauss 7d9052c059 [mod_openssl] compat with WolfSSL 2020-07-08 22:51:31 -04:00
Glenn Strauss b4036938de [mod_openssl] compat with LibreSSL 2020-07-08 22:51:31 -04:00
Glenn Strauss c93ecb32a5 [mod_openssl] disable session cache; prefer ticket
reduce memory usage and overhead

many TLSv1.2 implementations support TLSv1.2 session ticket extension
2020-07-08 22:51:31 -04:00
Glenn Strauss 83fb1aed6f [mod_gnutls] ssl.stek-file to specify encrypt key
difference from mod_openssl:

Admin should schedule an independent job to periodically
generate a new STEK before prior STEK lifetime expires.
Only one STEK is active at a time in mod_gnutls.

(more details in prior commit message for mod_openssl)
2020-07-08 22:51:31 -04:00
Glenn Strauss 3e2e8e6d29 [mod_mbedtls] ssl.stek-file to specify encrypt key
difference from mod_openssl:

Admin should schedule an independent job to periodically
generate a new STEK up to 2 times during key lifetime
(mbedtls internals store up to 2 keys)

(more details in prior commit message for mod_openssl)
2020-07-08 22:51:31 -04:00
Glenn Strauss 32a2145f67 [mod_openssl] ssl.stek-file to specify encrypt key
ssl.stek-file to specify session ticket encryption key (STEK)
If ssl.stek-file is specified, it overrides builtin STEK rotation.
STEK file is checked for changes (stat()) once every 64 seconds.

STEK file should be stored in non-persistent storage,
  e.g. /dev/shm/lighttpd/stek-file  (in memory)
with appropriate permissions set to keep stek-file from being
read by other users.  Where possible, systems should also be
configured without swap.

Admin should schedule an independent job to periodically
generate a new STEK up to 3 times during key lifetime
(lighttpd stores up to 3 keys)

format of binary file is:
   4-byte - format version (always 0; for use if format changes)
   4-byte - activation timestamp
   4-byte - expiration timestamp
  16-byte - session ticket key name
  32-byte - session ticket HMAC encrpytion key
  32-byte - session ticket AES encrpytion key

STEK file can be created with a command such as:
  dd if=/dev/random bs=1 count=80 status=none | \
    perl -e 'print pack("iii",0,time()+300,time()+86400),<>' \
    > STEK-file.$$ && mv STEK-file.$$ STEK-file

The above delays activation time by 5 mins (+300 sec) to allow file to
be propagated to other machines.  (admin must handle this independently)
If STEK generation is performed immediately prior to starting lighttpd,
admin should activate keys immediately (without +300).
2020-07-08 22:51:31 -04:00
Glenn Strauss 953fa4ba4d [mod_openssl] rotate session ticket encryption key
server ticket encryption key (STEK) rotation occurs every 28800 seconds
(8 hours) and upon lighttpd server restart.  While lighttpd is running,
(3) encryption keys are preserved, so tickets expire after 1 day.

If using lighttpd with multiple lighttpd workers, then restarting
lighttpd keeps the STEK in sync between lighttpd workers, though
restarting lighttpd three times a day might not be palatable.
Work is in progress to allow admin to supply new encryption keys.
2020-07-08 22:51:31 -04:00
Glenn Strauss 6d62a498a2 [core] more precise check for request stream flags 2020-07-08 22:51:31 -04:00
Glenn Strauss 7b9230ee51 [core] safe memset calls memset() through volatile
safe_memclear() calls memset() through volatile func ptr
2020-07-08 22:51:31 -04:00
Glenn Strauss 6f2dcc58cd [core] SecureZeroMemory() on _WIN32 2020-07-08 22:51:31 -04:00
Glenn Strauss 10c65e88f7 [mod_openssl] update default DH params
update default DH params to use RFC 7919 FFDHE2048 2048-bit group
if ssl.dh-file is not specified
2020-07-08 22:51:31 -04:00
Glenn Strauss 27dbac34a8 [mod_openssl] use openssl feature define for ALPN
use #ifdef TLSEXT_TYPE_application_layer_protocol_negotiation
rather than checking OPENSSL_VERSION_NUMBER >= 0x10002000 (1.0.2)
2020-07-08 22:51:31 -04:00
Glenn Strauss bdb3040f14 [mod_openssl] use SSL_OP_NO_RENEGOTIATION if avail
use [mod_openssl] SSL_OP_NO_RENEGOTIATION if available
(openssl 1.1.0 and later)
2020-07-08 22:51:31 -04:00
Glenn Strauss 3403382261 [mod_openssl] issue warning for deprecated options 2020-07-08 22:51:31 -04:00
Glenn Strauss d605c83a73 [mod_openssl] enable read-ahead, if set, after SNI
note: not disabling if enabled; extra blocks may already have been read
2020-07-08 22:51:31 -04:00
Glenn Strauss 1fc8a3e1f2 [core] sys-crypto-md.h w/ inline message digest fn
sys-crypto-md.h w/ inline message digest functions; shared code
2020-07-08 22:51:31 -04:00
Glenn Strauss c18f442a63 [multiple] add summaries to top of some modules 2020-07-08 22:51:31 -04:00
Glenn Strauss 10dbe38a92 [core] stricter parse of numerical digits
stricter parse of numerical digits for http status code, port num,
and a few other places.  (stricter parse than that of strtol())

content ranges are still parsed more loosely at points of use
2020-07-08 22:51:31 -04:00
Glenn Strauss 6876c16be0 [core] RFC-strict parse of Content-Length
augment simple strtoll() which allowed number to begin with '+'

This is not exploitable for HTTP Request Smuggling since lighttpd
mod_proxy sends "Connection: close" to backends, and other CGI-based
backends reconstitute CONTENT_LENGTH in the environment without '+'.

(thx Amit Klein, Safebreach)
2020-07-08 22:51:31 -04:00
Glenn Strauss acff179322 [mod_openssl] set chains from callback in 1.0.2+ (#2842)
mod_openssl leverages cert callback in openssl 1.0.2 and later

server certificate chain
  is now set in the cert callback

verify_store (for client certificate verification) (
  is now set in the cert callback

more carefully load sensitive files and clear temporary storage
  (with openssl 1.1.1 and later)

  "Lighttpd Returns Wrong Cert In Multi-cert Set-up"
2020-07-08 22:51:31 -04:00
Glenn Strauss 01b1f16b3f [mod_openssl] set cert from callback in 1.0.2+ (fixes #2842)
set server certificate from callback in openssl 1.0.2 and later

For existing versions of lighttpd, certificate selection influenced by
ssl.cipher-list which can be used to set server cipher order preference
(along with ssl.honor-cipher-order = "enable", which is the default)

  "Lighttpd Returns Wrong Cert In Multi-cert Set-up"
  "lighttpd uses wrong pem-file"
2020-07-08 22:51:31 -04:00
Glenn Strauss 2a5b7c648a [mod_openssl] rotate session ticket encryption key
server ticket encryption key (STEK) rotation occurs every 86400 seconds
and upon lighttpd server restart.  If using lighttpd with multiple
lighttpd workers, then restarting lighttpd keeps the STEK in sync
between lighttpd workers.
2020-07-08 22:51:31 -04:00
Glenn Strauss bf4054f8ec [mod_gnutls] GnuTLS option for TLS (fixes #109)

mod_gnutls supports most ssl.* config options supported by mod_openssl

  "GnuTLS support for the mod_ssl"
2020-07-08 22:51:31 -04:00
Glenn Strauss 68d626fa9f [mod_openssl] SSL_R_UNEXPECTED_EOF_WHILE_READING
report SSL_R_UNEXPECTED_EOF_WHILE_READING if debug.log-ssl-noise enabled

added in openssl 1.1.1e and reverted in 1.1.1f, but kept in 3.0 branch
2020-07-08 22:51:31 -04:00
Glenn Strauss b33b6f9121 [mod_openssl] do not log ECONNRESET unless debug
do not log ECONNRESET to errorlog unless debug.log-ssl-noise = "enable"
2020-07-08 22:51:31 -04:00
Glenn Strauss 446866333f [multiple] use *(unsigned char *) with ctypes 2020-07-08 22:51:31 -04:00
Glenn Strauss 455dc03778 [core] prefer getxattr() instead of get_attr()
(when lighttpd is built ./configure --with-attr)
2020-07-08 22:51:31 -04:00
Glenn Strauss cb753ec5b5 [mod_mbedtls] mbedTLS option for TLS

mod_mbedtls supports most ssl.* config options supported by mod_openssl

thx Ward Willats for the initial discussion and attempt in the comments
2020-07-08 22:51:31 -04:00
Glenn Strauss 7de51cc77b [core] add seed before openssl RAND_pseudo_bytes() 2020-07-08 19:54:30 -04:00
Glenn Strauss ce7840504d [mod_openssl] disable cert vrfy if ALPN acme-tls/1
disable client cert verification if ALPN "acme-tls/1"
2020-07-08 19:54:30 -04:00
Glenn Strauss f7bac374ee [mod_accesslog] process backslash-escapes in fmt
Process basic backslash-escapes in format string from lighttpd.conf
Supported sequences: \a \b \f \n \r \t \v
Other backslash-sequences are replaces with the char following backslash

(Apache mod_log_config supports \n and \t as special-cases)
2020-07-08 19:54:30 -04:00
Glenn Strauss f70912ee44 [mod_webdav] define PATH_MAX if not defined
(PATH_MAX does not appear to be defined on hurd-i386)
2020-07-08 19:54:30 -04:00
Glenn Strauss abe61d0445 [core] return EINVAL if stat_cache_get_entry w/o /
return EINVAL if stat_cache_get_entry() called on path w/o leading '/'
2020-07-08 19:54:30 -04:00
Glenn Strauss e6c47d6ecf [mod_cgi] remove redundant calls to set FD_CLOEXEC 2020-07-08 19:54:30 -04:00
Glenn Strauss 95b2774863 [core] skip excess close() when FD_CLOEXEC defined
do not run through close() loop after fork() and before execve()
when FD_CLOEXEC is defined
2020-07-08 19:54:30 -04:00
Glenn Strauss b28a3714c4 [multiple] ./configure --with-nettle to use Nettle
./configure --with-nettle to use Nettle crypto lib for algorithms,
instead of OpenSSL or wolfSSL.  Note: Nettle does not provide TLS.

  "How to use SHA-256 without OpenSSL?"
2020-07-08 19:54:30 -04:00
Glenn Strauss 0acb0f3604 [multiple] fix some cc warnings in 32-bit, powerpc 2020-07-08 19:54:30 -04:00
Glenn Strauss c752d4696e [multiple] correct misspellings in comments
  "Script for fixing spelling errors with codespell"
2020-07-08 19:54:30 -04:00
Glenn Strauss bcddbe186f [mod_proxy] stream request using HTTP/1.1 chunked (fixes #3006)
stream request body using HTTP/1.1 Transfer-Encoding: chunked

(Note: if backend proxy target does not support HTTP/1.1,
 then do not use = 1 or 2)

If not streaming to backend, collect request body
(now supporting Transfer-Encoding: chunked from client
 and then sending with Content-Length to backend)

  "Lighty returns HTTP 411 Length Required with proxy and streaming requests/reponses body"
2020-07-08 19:54:29 -04:00
Glenn Strauss 89082a12ca [config] blank server.tag if whitespace-only 2020-07-08 19:54:29 -04:00
Glenn Strauss bc89b98acb [build] fix build on MacOS X Tiger
fixes for Tiger build, but can be applied generally to all builds
also requires legacysupport PortGroup for *at functions

2020-07-08 19:54:29 -04:00
Glenn Strauss e1188e770e [mod_auth] "nonce_secret" option to validate nonce (fixes #2976)
"nonce_secret" option to validate nonce was generated by the server

Marginally hardens HTTP Digest Auth.  Necessary piece, but not
sufficient, to restrict re-use of nonce (mitigations for replay
or limiting nonce count reuse via nc=... are not implemented)

  "Digest auth nonces are not validated"
2020-07-08 19:54:29 -04:00
Glenn Strauss 9fdf24468d [doc] src/t/README 2020-07-08 19:54:29 -04:00
Glenn Strauss 41a268b805 [core] parse header label before end of line
parse header label for colon before checking end of line for whitespace
2020-07-08 19:54:29 -04:00
Glenn Strauss b04f0311f9 [core] perf: tighten some code in some hot paths 2020-07-08 19:54:29 -04:00
Glenn Strauss a200e0c204 [core] use uint32_t in http_header.[ch]
headers should never approach anywhere remotely close to > 4 GB
2020-07-08 19:54:29 -04:00
Glenn Strauss 1d6003e575 [core] mark some chunk.c funcs as pure 2020-07-08 19:54:29 -04:00
Glenn Strauss eae84b6a69 [core] http_status_append() short-circuit 200 OK 2020-07-08 19:54:29 -04:00
Glenn Strauss 17e0b6fe30 [tests] use buffer_eq_slen() for str comparison
(substitute buffer_eq_slen() for buffer_is_equal_string())
2020-07-08 19:54:29 -04:00
Glenn Strauss dd91c1459c [core] http_response_config()
split from http_response_prepare()
2020-07-08 19:54:29 -04:00
Glenn Strauss 3f4f934485 [core] http_response_comeback()
group HANDLER_COMEBACK logic in http_response_comeback() and call it
from places that reset state in order to (sometimes partially) reprocess
a request.  This includes error handler (server.error-handler),
r->handler_module when cgi.local-redir, and looping in
http_response_prepare() when modules make changes to the request and
return HANDLER_COMEBACK (e.g. mod_rewrite, mod_magnet, mod_cml)

Also, set r->conditional_is_valid closer to where elements are set
(and become valid for use in condition checks), and parse target
in http_request_parse() instead of http_response_prepare()
2020-07-08 19:54:29 -04:00
Glenn Strauss a0029b21a1 [core] remove r->uri.path_raw; generate as needed
(r->uri.path_raw previously duplicated from r->target, minus query-part)
2020-07-08 19:54:29 -04:00
Glenn Strauss 9aa0b3409e [mod_magnet] modify r->target with "uri.path-raw" 2020-07-08 19:54:29 -04:00
Glenn Strauss d013d0abd3 [core] http_request_parse_target()
http_request_parse_target() split from http_response_prepare()
2020-07-08 19:54:29 -04:00
Glenn Strauss 34285ed0fa [core] dispatch handler in handle_request func
dispatch handler in connection_handle_request()
instead of in http_response_prepare()
2020-07-08 19:54:29 -04:00
Glenn Strauss c8a1cba0c1 [tests] t/test_mod_userdir
create t/test_mod_userdir to replace sparse tests in tests/mod-userdir.t
remove tests/mod-userdir.t
2020-07-08 19:54:29 -04:00
Glenn Strauss 8b7e110973 [tests] stub out config funcs in test_mod_* 2020-07-08 19:54:29 -04:00
Glenn Strauss 9914bb297b [core] C99 restrict on some base funcs
buffer.[ch], chunk.[ch], request.[ch], sock_addr.[ch]
2020-07-08 19:54:29 -04:00
Glenn Strauss c687e01c98 [core] r->uri.path always set, though might be ""
(might be "" if <backend>.server sets "fix-root-scriptname" => "enable")
2020-07-08 19:54:29 -04:00
Glenn Strauss 7c7f8c467c [multiple] split con, request (very large change)
NB: r->tmp_buf == srv->tmp_buf (pointer is copied for quicker access)

NB: request read and write chunkqueues currently point to connection
    chunkqueues; per-request and per-connection chunkqueues are
    not distinct from one another
      con->read_queue  == r->read_queue
      con->write_queue == r->write_queue

NB: in the future, a separate connection config may be needed for
    connection-level module hooks.  Similarly, might need to have
    per-request chunkqueues separate from per-connection chunkqueues.
    Should probably also have a request_reset() which is distinct from
2020-07-08 19:54:29 -04:00
Glenn Strauss cc2134c88b [multiple] copy small struct instead of memcpy()
when patching config
2020-07-08 19:54:29 -04:00
Glenn Strauss 31d9495330 [core] store subrequest_handler instead of mode
store pointer to module in handler_module instead of con->mode id
2020-07-08 19:54:29 -04:00
Glenn Strauss eea7cd3c2f [core] store (plugin *) in p->data
2020-07-08 19:54:29 -04:00
Glenn Strauss aca9d45adf [core] move request state into (request_st *)
NB: in the future, a separate connection state may be needed for
    connection-level state (different from request state)
2020-07-08 19:54:29 -04:00
Glenn Strauss a22cdca1cb [core] move addtl request-specific struct members 2020-07-08 19:54:29 -04:00
Glenn Strauss 100dfaa3f3 [core] move plugin_ctx into (request_st *)
NB: in the future, a separate plugin_ctx may be needed for
    connection-level plugins to keep state across multiple requests
2020-07-08 19:54:29 -04:00
Glenn Strauss 8131e4396d [core] move addtl request-specific struct members 2020-07-08 19:54:29 -04:00
Glenn Strauss 1474be7859 [core] move addtl request-specific struct members 2020-07-08 19:54:29 -04:00
Glenn Strauss 6fe031ef37 [core] move request start ts into (request_st *)
move request start timestamps into (request_st *)
2020-07-08 19:54:29 -04:00
Glenn Strauss b157ee8dfa [mod_accesslog] log_access_record() fmt log record
separate func to append log record to buffer
2020-07-08 19:54:29 -04:00
Glenn Strauss 88ac4367f0 [core] (const connection *) in http_header_*_get() 2020-07-08 19:54:29 -04:00
Glenn Strauss 3648d2908e [core] t/test_request.c using (request_st *)
further isolate t/test_request.c tests to use (request_st *)
2020-07-08 19:54:29 -04:00
Glenn Strauss af5df35275 [core] rename content_length to reqbody_length
rename content_length to reqbody_length in request,
to more easily differentiate request body length
from response content_length
2020-07-08 19:54:29 -04:00
Glenn Strauss 8c46768d5b [core] remove unused request_st member 'request' 2020-07-08 19:54:29 -04:00
Glenn Strauss 6870b0f55b [core] pass (request_st *) to request.c funcs
instead of (struct connection *)
2020-07-08 19:54:29 -04:00
Glenn Strauss 668cb6418f [core] move request_st,request_config to request.h 2020-07-08 19:54:29 -04:00
Glenn Strauss 550faf988c [core] rename specific_config to request_config 2020-07-08 19:54:29 -04:00
Glenn Strauss af3313bfa5 [core] pass http_parseopts around request.c 2020-07-08 19:54:29 -04:00