Commit Graph

3074 Commits

Author SHA1 Message Date
Glenn Strauss a0ad558006 [mod_status] add additional HTML-encoding 2020-11-27 19:39:59 -05:00
Glenn Strauss 54922d61af [mod_fastcgi] move src/fastcgi.h into src/compat/ 2020-11-27 08:12:21 -05:00
Glenn Strauss 88433270c6 [mod_webdav] fallbacks if _ATFILE_SOURCE not avail
Note: filesystem access race conditions exist without _ATFILE_SOURCE
2020-11-27 08:12:21 -05:00
Glenn Strauss 7a0d94cd6f [mod_webdav] pass full path to webdav_unlinkat()
so that path can be removed from stat_cache
2020-11-27 08:12:21 -05:00
Glenn Strauss 09faa72f2f [mod_flv_streaming] use stat_cache_get_entry_open 2020-11-27 08:12:21 -05:00
Glenn Strauss edfc5f394e [core] consolidate chunk size checks 2020-11-27 08:12:21 -05:00
Glenn Strauss 35fa47d802 [core] quiet more request parse errs unless debug
quiet more request parse errors unless debug enabled with
  debug.log-request-header-on-error = "enable"

  "invalid character in URI -> 400 config?"
2020-11-27 08:12:21 -05:00
Glenn Strauss 647222b35c [core] check for __builtin_expect() availability 2020-11-27 08:12:21 -05:00
Glenn Strauss 730c932e3c [multiple] more forgiving config str to boolean (fixes #3036)
more consistent use of shared code config_plugin_value_tobool()

(thx tow-conf)

  "The on/off keywords in boolean configuration options is inconsistent, which might be misleading and error-prone."
2020-11-16 01:39:14 -05:00
Glenn Strauss c42b280583 [core] fix bug when HTTP/2 frames span chunks
fix inverted logic when HTTP/2 frames span chunkqueue chunks
2020-11-13 17:00:01 -05:00
Glenn Strauss f2b33e7520 [multiple] add back-pressure gw data pump (fixes #3033)
When = 0 (the default), the entire request
body is collected before engaging the backend.  For backends which
require data framing, this could lead to growth in memory use as large
requests were framed all at once.

Prefer to retain large request bodies in temporary files on disk and
frame in portions as write queue to backend drains below a threshold.

  "Memory Growth with PUT and full buffered streams"
2020-11-12 17:19:31 -05:00
Glenn Strauss 45aa1aa880 [mod_cgi] ensure tmp file open() before splice()
(bug on master branch)

With lighttpd defaults, including fully buffering request body, and
if request body > 1 MB, then multiple temporary files are used and
might not have open fd in chunkqueue.  This would result in failure
to send request body to CGI. (bug commited to master branch 1 month ago)
2020-11-12 01:44:19 -05:00
Glenn Strauss fc19558f96 [mod_cgi] fix crash if initial write to CGI fails
(bug on master branch)
2020-11-12 01:43:26 -05:00
Glenn Strauss 78ec2b5b68 [core] filter out duplicate modules
filter out modules duplicated in server.modules list
2020-11-11 20:57:41 -05:00
Glenn Strauss c16c6a8f8f [mod_alias] validate given order, not sorted order
(bug on master branch)

  "Debian Bullseye/sid arm64 - lighttp broken after update"
2020-11-11 19:06:51 -05:00
Glenn Strauss 5ec13918d3 [core] define SHA*_DIGEST_LENGTH macros if missing 2020-11-10 07:44:53 -05:00
Glenn Strauss 47aa6d4ac8 [core] http_response_buffer_append_authority()
make public func for benefit of external, third-party mod_authn_tkt
2020-11-10 06:10:27 -05:00
Glenn Strauss 291fd1e72e [core] accept "HTTP/2.0", "HTTP/3.0" from backends (#3031)
accept "HTTP/2.0" and "HTTP/3.0" NPH from naive non-proxy backends

(thx flynn)

  "uwsgi fails with HTTP/2"
2020-11-10 02:32:34 -05:00
Glenn Strauss 169d8d3608 [core] accept "HTTP/2.0", "HTTP/3.0" from backends (fixes #3031)
accept "HTTP/2.0" and "HTTP/3.0" NPH from naive non-proxy backends

(thx flynn)

  "uwsgi fails with HTTP/2"
2020-11-09 19:00:58 -05:00
Glenn Strauss c0e2667b71 [multiple] handle NULL val as empty in *_env_add (fixes #3030)
(bug on master branch; never released)

(thx flynn)

  "Fastcgi fails if server.tag is empty"
2020-11-09 18:20:13 -05:00
Glenn Strauss ff7cbcf5dd [mod_maxminddb] fix config validation typo
(bug on master branch; never released)

(thx maxentry)

  "maxminddb.env error 1.4.56"
2020-11-05 13:20:39 -05:00
Glenn Strauss 2a3fac7653 [mod_openssl] adjust LIBRESSL_VERSION_NUMBER check
libressl >= 0x3000000fL has SSL_set1_chain(), but not some other APIs
2020-11-05 01:17:30 -05:00
Glenn Strauss dce440602d [core] stat_cache preprocessor paranoia 2020-11-05 01:08:11 -05:00
Glenn Strauss 520bffcd24 [core] use struct kevent on stack in stat_cache
This alternative approach attempts to work around error:
  invalid application of 'sizeof' to incomplete type 'struct kevent'
seen in continuous integration (CI) autoconf build on FreeBSD VM
2020-11-05 00:48:41 -05:00
Glenn Strauss 57f450f199 [mod_openssl] add LIBRESSL_VERSION_NUMBER checks
add some additional LIBRESSL_VERSION_NUMBER checks for feature support
2020-11-04 22:47:36 -05:00
Glenn Strauss 8e1394ea49 [core] no graceful-restart-bg on OpenBSD, NetBSD
disable server.graceful-restart-bg on OpenBSD and NetBSD

kqueue is not inherited across fork, and OpenBSD and NetBSD do not
implement rfork() (implemented on FreeBSD and DragonFly)

lighttpd has not implemented rebuilding the kqueues after fork,
so server.graceful-restart-bg is disabled on OpenBSD and NetBSD.
2020-11-04 20:16:30 -05:00
Glenn Strauss 0b00b13a42 [core] use kqueue() instead of FAM/gamin on *BSD
Note: there have always been limitations with lighttpd stat_cache.[ch]
using FAM/gamin on *BSD via kqueue() as lighttpd stat_cache.[ch] only
monitors directories.  This kqueue() implementation also only monitors
directories and has limitations.

lighttpd stat_cache.[ch] is notified about additions and removals of
files within a monitored directory but might not be notified of changes
such as timestamps (touch), ownership, or even changes in contents
(e.g. if a file is edited through a hard link)

server.stat-cache-engine = "disable" should be used when files should
not be cached.  Full stop.  Similarly, "disable" is recommended if files
change frequently.  If using server.stat-cache-engine with any engine,
there are caching effects and tradeoffs.

On *BSD and using kqueue() on directories, any change detected clears
the stat_cache of all entries in that directory, since monitoring only
the directory does not indicate which file was added or removed.  This
is not efficient for directories containing frequently changed files.
2020-11-04 20:16:30 -05:00
Glenn Strauss 1efd74457b [core] cold func for gw_recv_response error case
2020-11-04 03:53:15 -05:00
Glenn Strauss 639f70a00f [core] set last_used on rd/wr from backend (fixes #3029)
  "mod_wstunnel kills child on disconnect after idle-timeout from connection start"
2020-11-04 03:53:15 -05:00
Glenn Strauss 3db556fde3 [mod_nss] update session ticket NSS devel comment
Update: NSS developer explains:
"The way that we currently operate is to tie the session key encryption
 to the server public key. Which only works if you have an RSA key
2020-11-04 03:53:15 -05:00
Glenn Strauss 7ce8b22cb2 [build] detect inotify header <sys/inotify.h> 2020-11-04 03:53:15 -05:00
Glenn Strauss 5c7173026f [core] use inotify in stat_cache.[ch] on Linux
use inotify in stat_cache.[ch] on Linux, replacing FAM/gamin
2020-11-04 03:53:15 -05:00
Glenn Strauss 64aff80d86 [build] prepend search for lua version 5.4 2020-10-31 00:55:44 -04:00
Glenn Strauss 89cd3d8b9d [meson] add matching -I for lua lib version 2020-10-31 00:55:44 -04:00
Glenn Strauss 07622251b0 [build] option to use system-provided libxxhash
2020-10-29 19:30:45 -04:00
Glenn Strauss 28b086aa26 [core] use system xxhash lib if available 2020-10-29 17:20:25 -04:00
Glenn Strauss bcac9b5785 [core] set NSS_VER_INCLUDE after crypto lib select 2020-10-29 16:49:35 -04:00
Glenn Strauss 8187e98897 [multiple] include wolfssl/options.h after select
include wolfssl/options.h crypto lib config
after selecting crypto lib to use

wolfSSL does not prefix its defines with a wolfSSL-specific namespace
(so we would like to avoid unnecessarily polluting preproc namespace)

This commit further isolates wolfSSL after split from mod_openssl.
Cleans up some preprocessor logic that was put in place when using
the wolfSSL compatibility layer for openssl, before creating a
dedicated mod_wolfssl.
2020-10-29 16:42:55 -04:00
Glenn Strauss 6fb63fa8d6 [multiple] include mbedtls/config.h after select
include mbedtls/config.h crypto lib config
after selecting crypto lib to use
2020-10-29 16:41:27 -04:00
Glenn Strauss 441c95c697 [multiple] consistent order for crypto lib select 2020-10-29 16:39:56 -04:00
Glenn Strauss e726a41a74 [core] adjust wolfssl workaround for another case
adjust wolfssl types.h workaround for another edge case
2020-10-29 05:45:18 -04:00
Glenn Strauss 78b13b610c [core] move misplaced error trace to match option
(thx grohne)
2020-10-29 05:34:43 -04:00
Glenn Strauss e9590277ca [core] workaround fragile code in wolfssl types.h
workaround fragile code in wolfssl/wolfcrypto/types.h

Including header blows up compile in 32-bit when lighttpd meson build
in OpenWRT on a 32-bit platform generates lighttpd config.h containing
define of SIZEOF_LONG, but not SIZEOF_LONG_LONG, and the wolfssl types.h
flubs and fails to choose an enum value used by a macro that is unused
by most consumers of the wolfssl header.
2020-10-29 05:06:24 -04:00
Glenn Strauss 31fc3a0773 [TLS] server.feature-flags "ssl.session-cache"
disabled by default, but can be enabled
(session tickets should be preferred)

applies to mod_openssl, mod_wolfssl, mod_nss

session cache is not currently implemented in mod_mbedtls or mod_gnutls
2020-10-29 01:05:55 -04:00
Glenn Strauss 1d27391c29 [mod_mbedtls] wrap addtl code in preproc defines
wrap additional code in preprocessor defines to check if enabled in lib
2020-10-28 22:58:47 -04:00
Glenn Strauss 3353ff2024 [build] adjust for use by OpenWRT 2020-10-28 22:58:47 -04:00
Glenn Strauss babfb43873 [build] WITHOUT_LIB_CRYPTO option in code
(not (yet?) an end-user option in the build system)
(If extended to build system, build system should also unset CRYPTO_LIB)

If WITHOUT_LIB_CRYPTO is defined in sys-crypto.h, then non-TLS modules
will have access to MD5() and SHA1() built with lighttpd (algo_md5.[ch]
and algo_sha1.[ch]), but not to other message digest algorithms.

As of this commit, this affects only mod_secdownload with SHA256 digest
and mod_auth* modules using HTTP Digest Auth with digest=SHA-256, which
is not currently well-supported by client browers (besides Opera)
2020-10-27 16:47:33 -04:00
Glenn Strauss 31a7f0d43c [build] detect nss3/nss.h or nss/nss.h for NSS 2020-10-27 16:40:08 -04:00
Glenn Strauss 75c29505be [build] CMake use pkg_check_modules() w/ wolfssl 2020-10-27 13:01:01 -04:00
Glenn Strauss 07f40eaf9a [build] CMake mod_openssl, mod_wolfssl can coexist 2020-10-27 12:37:48 -04:00
Glenn Strauss e130124892 [build] add --with-brotli to 2020-10-27 12:32:10 -04:00
Glenn Strauss 1f1b3bcc55 [mod_secdownload] fix compile w/ NSS on FreeBSD
(thx dirk)

nss/alghmac.h might not exist
2020-10-26 22:24:24 -04:00
Glenn Strauss 988fa8ecfa [mod_wolfssl] need to build --enable-alpn for ALPN
need to build wolfSSL library with --enable-alpn for ALPN
even if already building wolfSSL library with --enable-openssall

ALPN is required by the HTTP/2 specification
2020-10-26 22:21:16 -04:00
Glenn Strauss 17d8d9c919 [mod_wolfssl] cripple SNI if not built OPENSSL_ALL
crippled functionality if wolfssl library not built --enable-opensslall
* SNI not handled since SNI callbacks are disabled in wolfSSL library
  unless the wolfSSL library is built with --enable-openssall

  This means that there is only one certificate per listening socket --
  no certificate selection based on server name indication (SNI)
  and is additionally a violation of the HTTP/2 specification,
  which requires SNI.

slightly reduced functionality if wolfssl not built --enable-opensslall
* disable client certificate verification (error out if in lighttpd.conf)
2020-10-26 22:12:39 -04:00
Glenn Strauss 1221bd6e40 [mod_wolfssl] use more wolfssl/options.h defines 2020-10-26 20:26:24 -04:00
Glenn Strauss a1f6d1322a [build] mark dependencies on crypto lib for MD5()
(lighttpd base executable depends on crypto lib for rand functionality,
 so the crypto library was already being loaded -- no missing symbols)
2020-10-26 13:53:07 -04:00
Glenn Strauss 1fad70f4f7 [core] STAILQ_* -> SIMPLEQ_* on OpenBSD
(thx brad)
2020-10-26 11:29:40 -04:00
Glenn Strauss 48384c7e2f [core] fix (startup) mem leaks in configparser.y
(thx stbuehler)
2020-10-24 17:06:50 -04:00
Glenn Strauss 949662d27e [multiple] add some missing config cleanup
(thx stbuehler)
2020-10-24 16:08:21 -04:00
Glenn Strauss 55fb46f695 [mod_accesslog] update defaults after cycling log
(thx avij)

must update the cached copy of global scope config after cycling log.
Although (accesslog_st *) is modified in-place, the log_access_fd member
of (accesslog_st *) is copied into the cache and must be updated after
cycling logs in the global scope.
2020-10-24 14:38:47 -04:00
Glenn Strauss 61f7d531eb [mod_mbedtls] newer mbedTLS vers support TLSv1.3 2020-10-24 02:03:05 -04:00
Glenn Strauss 0a2aab88d2 [core] silence coverity warnings (another try) 2020-10-23 23:14:06 -04:00
Glenn Strauss ad62991a5b [mod_webdav] define _NETBSD_SOURCE on NetBSD
NetBSD dirent.h improperly hides fdopendir() (POSIX.1-2008) declaration
which should be visible w/ _XOPEN_SOURCE 700 or _POSIX_C_SOURCE 200809L
2020-10-23 21:29:43 -04:00
Glenn Strauss f65b054887 [mod_nss] more nss includes fixes 2020-10-22 01:34:25 -04:00
Glenn Strauss 033209393e [multiple] test for nss includes
some distro packages deploy NSS includes under nss/, others nss3/
(and similar for nspr/ vs nspr4/)
2020-10-22 00:48:40 -04:00
Glenn Strauss 0158c7f0fd [build] remove svnversion from versionstamp rule 2020-10-21 20:56:18 -04:00
Glenn Strauss 23fdff645a [core] init NSS lib for basic crypto algorithms
basic algorithms fail if NSS library has not been init'd (WTH)

lighttpd defers initialization of rand and crypto until first use
to attempt to avoid long, blocking init at startup while waiting
for sufficient system entropy to become available
2020-10-21 18:18:40 -04:00
Glenn Strauss 9868d3b348 [core] add missing declaration for NSS rand
(bug on master branch; never released)
2020-10-21 17:05:24 -04:00
Glenn Strauss 2209c0cf6e [core] use inline funcs in sys-crypto-md.h
use inline funcs in sys-crypto-md.h for consistency
and to avoid compiler warnings when result is ignored
2020-10-21 00:57:24 -04:00
Glenn Strauss f98dff9bc3 [mod_mbedtls] quiet unused variable warning 2020-10-20 23:16:00 -04:00
Glenn Strauss 8f7019adf0 [core] fix fd sharing when splitting file chunk
(bug on master branch; never released)

fix fd sharing in chunkqueue_steal_partial_file_chunk()
2020-10-20 16:53:51 -04:00
Glenn Strauss 0f82722f5f [core] silence coverity warnings (another try) 2020-10-20 16:22:38 -04:00
Glenn Strauss fb7e5a7832 [core] silence coverity warnings in ls-hpack
The code originates from
and is explicitly documented as not needing to be initialized.

2020-10-20 16:04:48 -04:00
Glenn Strauss af04e0b0e1 [core] silence coverity warnings (false positives) 2020-10-20 15:29:25 -04:00
Glenn Strauss 1b74c50854 [core] always lseek() with shared fd
always lseek() with shared fd; remove optim to skip with offset = 0
2020-10-20 11:51:48 -04:00
Glenn Strauss 019c513819 [multiple] use http_chunk_append_file_ref()
use http_chunk_append_file_ref() and http_chunk_append_file_ref_range()

reduce resource usage (number of fds open) by reference counting open
fds to files served, and sharing the fd among FILE_CHUNKs in responses
2020-10-20 11:51:48 -04:00
Glenn Strauss 9078cc4ce8 [core] http_chunk_append_file_ref_range()
http_chunk_append_file_ref() and http_chunk_append_file_ref_range()
to take stat_cache_entry ref and append FILE_CHUNK
2020-10-20 11:51:48 -04:00
Glenn Strauss 6be2bd35a1 [core] FILE_CHUNK can hold stat_cache_entry ref 2020-10-20 11:51:48 -04:00
Glenn Strauss 7f8ab9dd29 [core] stat_cache_entry reference counting
future: should probably create fd cache separate from stat_cache,
        perhaps along w/ http-specific fields like etag and content_type
        and maybe even mmap
2020-10-20 11:51:48 -04:00
Glenn Strauss e057c5413b [mod_deflate] use large mmap chunks to compress
use large chunks since server blocks while compressing, anyway
(mod_deflate is not recommended for large files)
2020-10-20 11:51:47 -04:00
Glenn Strauss fc6612d7c7 [core] minimize pause during graceful restart
minimize pause during graceful restart for server.max-worker = 0 case

The previous generation continues to accept new connections until the
restarted parent signals that the restarted server is ready to accept
new connections, and so the previous server should gracefully shutdown.

This does not apply in the case of multiple workers.

When there are multiple workers, they receive SIGINT to gracefully shut
down and stop accepting new connections.  While the listen sockets are
kept open (and not closed and reopened), there is a small pause while
the parent process restarts before it begins accepting new connections
from the listen backlog.

Note: there is a window during restart during which lighttpd may exit
if it receives certain signals before it sets up signal handlers.
future: might block signals (sigprocmask()) during restart, but if that
is done, then care must be taken to unblock signals in restarted server
as soon as signal handlers are set up and before any other children are
created, e.g. by modules, or else signals must be explicitly unblocked
in children.  Also, during command line and config file processing,
signals would be blocked, too, which might not be ideal.
2020-10-19 21:40:14 -04:00
Glenn Strauss d6f692f1c2 [core] config_plugin_value_to_int32() 2020-10-19 21:40:14 -04:00
Glenn Strauss 3d8dcbdd43 [mod_openssl] use newer openssl 3.0.0 func
replace X509_STORE_load_locations() with X509_STORE_load_file()
2020-10-19 21:40:14 -04:00
Glenn Strauss 1cc81b703f [mod_webdav] update defaults after worker_init
update defaults after worker_init for config options in global scope

(bug on master branch; never released)
2020-10-19 21:40:14 -04:00
Glenn Strauss f973b87ca2 [mod_webdav] add missing update to cq accounting
(bug on master branch; never released)
2020-10-19 21:40:14 -04:00
Glenn Strauss d71c1d47ce [core] modify use of getrlimit() to not be fatal
modify use of getrlimit() to not be fatal if an error occurs

Some container configurations might cause getrlimit() and setrlimit() to
fail.  lighttpd will issue error trace if this occurs, but will now

  "aarch64: Use architecture specific header to general syscalls"
  "Rasperry Docker PiHole / "Starting lighttpd" Hanging"
  "Pi-hole runs on podman"
2020-10-19 21:40:14 -04:00
Glenn Strauss 3f1a12e5fb [multiple] remove chunk file.start member
c->offset is now offset into file
c->file.length is end of octets to send (end pos + 1)
c->file.length - c->offset is num of octets to send
2020-10-19 21:40:14 -04:00
Glenn Strauss d865d8c330 [TLS] ignore empty "CipherString" in ssl-conf-cmd
e.g. ssl.openssl.ssl-conf-cmd = ("CipherString" => "")
2020-10-19 21:40:14 -04:00
Glenn Strauss fe02111888 [multiple] stat_cache_path_stat() for struct st
stat_cache_path_stat() for cached (struct st *)
2020-10-19 21:40:14 -04:00
Glenn Strauss a46f519eb2 [multiple] use NSS crypto if no other crypto avail
use NSS crypto if no other crypto avail, but NSS crypto is available

"NSS crypto support" is not included in tests/
due to NSS libraries (freebl3) lacking public export for HMAC funcs
2020-10-19 21:40:14 -04:00
Glenn Strauss bdb5fb26a8 [wolfssl] wolfSSL_CTX_set_mode differs from others
wolfSSL_CTX_set_mode() differs from openssl SSL_CTX_set_mode().
wolfSSL_CTX_set_mode() takes a single flag at a time and has
sparse flag support (small number of recognized flags)
2020-10-19 21:40:14 -04:00
Glenn Strauss cf8cefceb6 [core] sys-crypto-md.h consistent interfaces
return values for sys-crypto-md.h interfaces

While some library implementations do not fail and have no return value,
others might fail on memory allocation or on failure to communicate with
an external or dedicated engine or device, e.g. which might store a
private key.

future: lighttpd callers of sys-crypto-md.h do not currently expect
        or check for errors from these digest functions, but should
        consider doing so.
2020-10-19 21:40:14 -04:00
Glenn Strauss 7553ef7f44 [multiple] openssl 3.0.0 digest interface migrate
provide implementations for conventional digest interfaces
but use the newer openssl digest interfaces under the hood

It is baffling that the openssl library -- with *thousands* of public
interfaces -- does not provide these, and suggests that openssl
developers do not frequently write apps which utilize these interfaces.
2020-10-19 21:40:14 -04:00
Glenn Strauss e6a87a894d [mod_openssl] merge ssl_tlsext_ticket_key_cb impls 2020-10-19 21:40:14 -04:00
Glenn Strauss cc6a76be12 [mod_openssl] construct OSSL_PARAM on stack 2020-10-19 21:40:14 -04:00
Glenn Strauss ebafd914e7 [mod_openssl] migrate ticket cb to openssl 3.0.0
migrate ticket cb to openssl 3.0.0 interface
2020-10-19 21:40:14 -04:00
Glenn Strauss e1bcdb5e9f [multiple] ignore openssl 3.0.0 deprecation warns
quiet openssl 3.0.0 deprecation warnings for
low-level digest algorithm interfaces

future: migrate to openssl interfaces:
  EVP_DigestInit_ex(3), EVP_DigestUpdate(3), and EVP_DigestFinal_ex(3)

2020-10-19 21:40:14 -04:00
Glenn Strauss a3ccc46be1 [mod_cgi] use splice() to send input to CGI
use splice(), if available (Linux), to send input from temp files to CGI
2020-10-19 21:40:14 -04:00
Glenn Strauss e18da93e44 [core] reuse chunkqueue_compact_mem*() 2020-10-19 21:40:14 -04:00
Glenn Strauss 9c25581d6f [core] alloc optim reading file, sending chunked
avoid potential double-copy due to not enough space for final '\0'
in http_chunk_append_read_fd_range() if read size is exactly multiple
of 8k and sending chunked response
2020-10-19 21:40:14 -04:00
Glenn Strauss 6d76f7ce49 [core] fix chunkqueue_compact_mem w/ partial chunk
(bug on master branch; never released)
2020-10-19 21:40:14 -04:00
Glenn Strauss 8abbf621d1 [mod_fastcgi] decode chunked is cold code path
decode chunked from FastCGI backend is cold code path
2020-10-19 21:40:14 -04:00
Glenn Strauss 96abd9cfb8 [core] coalesce nearby ranges in Range requests
Range requests must be HTTP/1.1 or later (not HTTP/1.0)
2020-10-19 21:40:14 -04:00
Glenn Strauss 66d1ec485c [core,mod_deflate] leverage cache of open fd
leverage simple cache of open file in stat_cache
(use stat_cache_get_entry_open())

future: reference count fd instead of dup()
  (requires extending chunkqueue interfaces)
2020-10-19 21:40:14 -04:00
Glenn Strauss d8e5e21eb7 [core] stat_cache_get_entry_open()
simple interface to cache open file by extending struct stat_cache_entry

future: should probably create fd cache separate from stat_cache,
        perhaps along w/ http-specific fields like etag and content_type
2020-10-19 21:40:14 -04:00
Glenn Strauss e99126074c [core] pass open fd to http_response_parse_range 2020-10-13 22:31:10 -04:00
Glenn Strauss 6219b861ce [core] http_response_parse_range() const file sz 2020-10-13 22:31:10 -04:00
Glenn Strauss cd2ff4202d [core] remove unused r->content_length
(was parsed Content-Length sent by backend)
(might add back in future along with stricter validation of field value)
2020-10-13 22:31:10 -04:00
Glenn Strauss 55e9f082a9 [core] limit num ranges in Range requests
parse Range header prior to constructing ranged response
2020-10-13 22:31:10 -04:00
Glenn Strauss 2dc3d5faac [mod_gnutls] quiet CLOSE_NOTIFY after conn reset
do not log error after connection reset
2020-10-13 22:31:10 -04:00
Glenn Strauss 496cd8ff44 [mod_mbedtls] quiet CLOSE_NOTIFY after conn reset
do not log error after connection reset
2020-10-13 22:31:10 -04:00
Glenn Strauss f846a392d5 [multiple] use stat_cache_path_isdir() 2020-10-13 22:31:10 -04:00
Glenn Strauss 7d368cd7a5 [core] stat_cache_path_isdir() 2020-10-13 22:31:07 -04:00
Glenn Strauss 14eb97f1b3 [core] dup FILE_CHUNK fd when splitting FILE_CHUNK 2020-10-11 12:19:27 -04:00
Glenn Strauss 2b4166fb74 [core] fdevent_dup_cloexec() 2020-10-11 12:19:27 -04:00
Glenn Strauss ad22eebd23 [mod_ssi] use stat_cache_open_rdonly_fstat() 2020-10-11 12:19:27 -04:00
Glenn Strauss b7370a6d46 [core] save errno around close() if fstat() fails
(should not happen, since file was just opened)
2020-10-11 12:19:27 -04:00
Glenn Strauss e2c832122b [mod_deflate] fix potential NULL deref in err case
(bug on master branch; never released)
2020-10-11 12:19:27 -04:00
Glenn Strauss 4d6d1e790a [multiple] use light_btst() for hdr existence chk 2020-10-11 12:19:27 -04:00
Glenn Strauss bd8edb51d0 [core] allow symlinks under /dev for rand devices
(fix code to match comment)
2020-10-11 12:19:27 -04:00
Glenn Strauss 874707cd66 [TLS] use fdevent_load_file_bytes() for STEK file
remove direct dependency on <unistd.h> from lighttpd TLS modules
2020-10-11 12:19:27 -04:00
Glenn Strauss 771ebd39a9 [core] fdevent_load_file_bytes()
wrapper to load defined number of bytes from file at given offset
2020-10-11 12:19:27 -04:00
Glenn Strauss ee1f7af24d [core] be more precise checking streaming flags 2020-10-11 12:19:27 -04:00
Glenn Strauss f7fc8d8d75 [core] connection_joblist global
con->srv->joblist is expensive to traverse when cache is cold
and called from connection_handle_fdevent(); var made a global
2020-10-11 12:19:27 -04:00
Glenn Strauss 01f90c58ef [core] walk queue in connection_write_chunkqueue()
walk chunkqueue up to first FILE_CHUNK (if present)
This may incur memory load misses for pointer chasing, but effectively
preloads part of the chunkqueue, something which used to be a side
effect of a previous (less efficient) version of chunkqueue_length()
which walked the entire chunkqueue (on each and every call).  The loads
here make a measurable difference in performance in underlying call to
2020-10-11 12:19:27 -04:00
Glenn Strauss f1136f7897 [core] short-circuit connection_write_throttle()
short-circuit connection_write_throttle() when throttling not enabled
2020-10-11 12:19:27 -04:00
Glenn Strauss 7c1e81299f [core] handle unexpected EOF reading FILE_CHUNK
(replace existing check which suffered from ToC-ToU race condition)
enhances logic from 2015 commit 593599f1 and avoids repeated fstat()
checks when sending large files

For mmap(), lighttpd catches SIGBUS if file is (externally) truncated
and lighttpd attempts to access bytes in a read-only mapping more than
a memory page boundary following the end of the file.

For sendfile(), lighttpd returns an error if sendfile() reports no error
and that no bytes have been sent after lighttpd attempts to send a
non-zero number of bytes.
2020-10-11 12:19:27 -04:00
Glenn Strauss a8398e4596 [core] defer handling FDEVENT_HUP and FDEVENT_ERR
defer handling FDEVENT_HUP and FDEVENT_ERR to after processing
(connection *) in order to have a chance to read data in kernel
socket buffers
2020-10-11 12:19:27 -04:00
Glenn Strauss 52d9b0da88 [core] server_run_con_queue()
func to run queued (connection *) jobs
2020-10-11 12:19:27 -04:00
Glenn Strauss a3001b968f [core] propagate state after writing
(perf) avoid reload miss after partial write of chunkqueue data
2020-10-11 12:19:27 -04:00
Glenn Strauss 81029b8b51 [multiple] inline chunkqueue where always alloc'd
inline struct chunkqueue where always allocated in other structs

(memory locality)
2020-10-11 12:19:27 -04:00
Glenn Strauss 8b382a81c7 [multiple] use sock_addr_get_family in more places 2020-10-11 12:19:27 -04:00
Glenn Strauss 86e5f09062 [core] perf adjustments to avoid load miss 2020-10-11 12:19:27 -04:00
Glenn Strauss ed297e7e67 [core] lighttpd -1 supports pipes (e.g. netcat)
lighttpd -1 (one-shot mode) now supports pipes (e.g. with netcat)

  "Is there anyway to run lighttpd in the command line?"
2020-10-11 12:19:27 -04:00
Glenn Strauss 94c4c63773 [core] graceful shutdown timeout option
server.feature-flags += ("server.graceful-shutdown-timeout" => 10)

After receiving SIGINT or SIGUSR1, lighttpd will gracefully shutdown,
waiting for existing connections to complete.  In the case of SIGUSR1,
this wait occurs before restarting lighttpd.  The default timeout is
none (unlimited).

When "server.graceful-shutdown-timeout" option is set, it defines the
number of seconds that lighttpd will wait for existing connections to
complete before shutting down the connection.

Sites which expect large uploads or downloads, or those with very slow
clients, might want to set a much longer timeout, e.g 60 seconds

For more immediate graceful restarts, while still allowing existing
connections time to complete, sites should additionally consider
whether or not
  server.feature-flags += ("server.graceful-restart-bg" => "enable")
is appropriate and compatible with their lighttpd.conf settings
2020-10-11 12:19:27 -04:00
Glenn Strauss 76bd8bba9a [mod_ssi] init status var before waitpid() 2020-10-11 12:19:27 -04:00
Glenn Strauss 352d5d776d [core] graceful and immediate restart option
graceful and (nearly) immediate lighttpd restart option

For *some* configurations, it *may* be safe to background the current
lighttpd server (or workers) to continue processing active requests
and, in parallel, to start up a new lighttpd server with a new
configuration.  For other configurations, doing so might not be safe!

Therefore, this option must be explicitly configured to enable:
  server.feature-flags += ("server.graceful-restart-bg" => "enable")
  server.systemd-socket-activation = "enable"

Along with enabling server.feature-flags "server.graceful-restart-bg",
enabling server.systemd-socket-activation allows transfer of open
listening sockets to the new lighttpd server instance, and occurs
without closing the listening sockets and without destroying the
kernel listen backlog queue on the socket.

Safe configurations may include lighttpd.conf which connect to
standalone backend daemons, e.g. proxying to other servers,
including PHP-FPM backends.

Unsafe configurations include lighttpd.conf which use "bin-path" option
in *.server configs, instructing lighttpd to execute the backends.
Using the graceful-and-immediate-restart option is likely *unsafe* if
the backend daemon expects only one instance of itself to run at a time.

Current implementation of graceful and immediate restart option keeps
the backgrounded lighttpd in the same process group, so that subsequent
SIGINT or SIGTERM will shut down both the new and the backgrounded
servers.  (An alternative option (commented out in the code) is to
background and detach from the new lighttpd process.)  Regardless,
existing subprocesses, such as CGI, remain in original process group.
As a result, the new lighttpd server may receive SIGCHLD for unknown
processes inherited from the old server, which the new lighttpd server
will reap and discard.  The original lighttpd server, now a child, will
be unable to detect exit or reap and report status on those pre-existing

Graceful restart is triggered in lighttpd by sending lighttpd SIGUSR1.
If lighttpd is configured with workers, then SIGINT (not SIGUSR1) is
sent to the process group, including other processes started by
lighttpd, e.g. CGI.  To work well with graceful restart, CGI scripts and
other processes should trap SIGINT (and SIGUSR1 for good measure).
Long-running scripts may want to checkpoint and close, e.g. a CGI script
implementing a long-running websocket connection.
2020-10-11 12:19:27 -04:00
Glenn Strauss f779d354a2 [core] config_plugin_value_tobool() accept "0","1" 2020-10-11 12:19:27 -04:00
Glenn Strauss 1a64c9e2f7 [core] reuse r->start_hp.tv_sec for r->start_ts
(remove duplicated field from (request_st *))
2020-10-11 12:19:27 -04:00
Glenn Strauss 660d719a2a [multiple] code reuse chunkqueue_peek_data()
code reuse of chunkqueue_peek_data() and chunkqueue_read_data()
2020-10-11 12:19:27 -04:00
Glenn Strauss 29e66e70e2 [mod_magnet] magnet.attract-response-start-to

add option to run lua scripts in lighttpd response start hook

allows for response header manipulation

new params provide read-only access:

allows for content manipulation if the response body is complete

The HTTP response status can be accessed in lua via
lighty.env["response.http-status"] and should be checked, as
appropriate, prior to body manipulation.  The value is non-zero
in response start hook (magnet.attract-response-start-to), but is
likely to be 0 in scripts run from other lighttpd hooks earlier in
request processing,
e.g. magnet.attract-raw-url-to or magnet.attract-physical-path-to

Caller should check lighty.env["response.body-length"]
is a smaller and sane amount to read into memory and copy
a second time into lua data structures.  The value is lua nil
if the response body is not yet complete (or if it is >= 2GB-1)

Loading the response body (and all mod_magnet lua scripts) are
executed serially (blocking) in lighttpd, so its use is highly
discouraged on large files.  The body can be accessed in lua via
lighty.env["response.body"] if the response body is complete.
(recommended config option: = 0 (default)
 if mod_magnet scripts must process the response body)

Modifying HTTP response status and response body has not changed
and is achieved by setting lua script return value and modifying
the lighty.content lua table.

(note: mod_magnet, mod_setenv, mod_deflate, mod_expire have their
 response start hooks run in the order listed in server.modules)
2020-10-11 12:19:27 -04:00
Glenn Strauss 16a70b9253 [multiple] TLS modules use chunkqueue_peek_data() 2020-10-11 12:19:27 -04:00
Glenn Strauss 6c68e14940 [core] chunkqueue_{peek,read}_data(), squash
chunkqueue_peek_data(), chunkqueue_read_data(), chunkqueue_read_squash()
shared code for chunkqueue manipulation
2020-10-11 12:19:27 -04:00
Glenn Strauss 362be7b3bb [core] relay 1xx from backend over HTTP/1.1
relay 1xx from backend over HTTP/1.1, e.g. 103 Early Hints
(if client is connected using HTTP/1.1)

enabled by default unless disabled in lighttpd.conf with:
  server.feature-flags += ( "server.h1-discard-backend-1xx" = "enable" )

Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response.  Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request. Security Considerations

  An HTTP Status Code for Indicating Hints (103 Early Hints)
2020-10-11 12:19:27 -04:00
Glenn Strauss 869c778aa7 [core] relay 1xx from backend over HTTP/2
relay 1xx from backend over HTTP/2, e.g. 103 Early Hints
(if client is connected using HTTP/2)

enabled by default unless disabled in lighttpd.conf with:
  server.feature-flags += ( "server.h2-discard-backend-1xx" = "enable" )

Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response.  Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request. Security Considerations

  An HTTP Status Code for Indicating Hints (103 Early Hints)
2020-10-11 12:19:26 -04:00
Glenn Strauss 10d9d14633 [core] reload c after chunkqueue_compact_mem()
reload c = cq->first after calling chunkqueue_compact_mem()
2020-10-11 12:19:26 -04:00
Glenn Strauss edbfe36563 [core] support multiple 1xx responses from backend
support multiple 1xx intermediate responses from backends

Currently, all 1xx responses from backends are discarded.
In the future, these 1xx responses may be forwarded to the client
(when lighttpd also configured = 1 or = 2)
2020-10-11 12:19:26 -04:00
Glenn Strauss 7a3098398c [core] connection_check_expect_100()
isolate code to check for Expect: 100-continue
(separate out from code to read request body)
2020-10-11 12:19:26 -04:00
Glenn Strauss a330746f06 [TLS] error if inherit empty TLS cfg from globals
error if $SERVER["socket"] inherits empty TLS config from global scope
and ssl.engine = "enable" in the $SERVER["socket"]
2020-10-11 12:19:26 -04:00
Glenn Strauss 5ba175269a [core] skip ls-hpack decode work unused by lighttpd
define LSHPACK_DEC_CALC_HASH 0 to skip work whose results are not used
by lighttpd
2020-10-11 12:19:26 -04:00