(experimental)
add option to run lua scripts in lighttpd response start hook
allows for response header manipulation
new params provide read-only access:
lighty.env["response.http-status"]
lighty.env["response.body-length"]
lighty.env["response.body"]
allows for content manipulation if the response body is complete
The HTTP response status can be accessed in lua via
lighty.env["response.http-status"] and should be checked, as
appropriate, prior to body manipulation. The value is non-zero
in response start hook (magnet.attract-response-start-to), but is
likely to be 0 in scripts run from other lighttpd hooks earlier in
request processing,
e.g. magnet.attract-raw-url-to or magnet.attract-physical-path-to
Caller should check lighty.env["response.body-length"]
is a smaller and sane amount to read into memory and copy
a second time into lua data structures. The value is lua nil
if the response body is not yet complete (or if it is >= 2GB-1)
Loading the response body (and all mod_magnet lua scripts) are
executed serially (blocking) in lighttpd, so its use is highly
discouraged on large files. The body can be accessed in lua via
lighty.env["response.body"] if the response body is complete.
(recommended config option: server.stream-response-body = 0 (default)
if mod_magnet scripts must process the response body)
Modifying HTTP response status and response body has not changed
and is achieved by setting lua script return value and modifying
the lighty.content lua table.
(note: mod_magnet, mod_setenv, mod_deflate, mod_expire have their
response start hooks run in the order listed in server.modules)
relay 1xx from backend over HTTP/1.1, e.g. 103 Early Hints
(if client is connected using HTTP/1.1)
enabled by default unless disabled in lighttpd.conf with:
server.feature-flags += ( "server.h1-discard-backend-1xx" = "enable" )
Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response. Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request.
https://tools.ietf.org/html/rfc8297#section-3 Security Considerations
x-ref:
An HTTP Status Code for Indicating Hints (103 Early Hints)
https://tools.ietf.org/html/rfc8297
relay 1xx from backend over HTTP/2, e.g. 103 Early Hints
(if client is connected using HTTP/2)
enabled by default unless disabled in lighttpd.conf with:
server.feature-flags += ( "server.h2-discard-backend-1xx" = "enable" )
Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response. Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request.
https://tools.ietf.org/html/rfc8297#section-3 Security Considerations
x-ref:
An HTTP Status Code for Indicating Hints (103 Early Hints)
https://tools.ietf.org/html/rfc8297
support multiple 1xx intermediate responses from backends
Currently, all 1xx responses from backends are discarded.
In the future, these 1xx responses may be forwarded to the client
(when lighttpd also configured server.stream-response-body = 1 or = 2)
(optional addition to (data_string *), used by http_header.[ch])
extend (data_string *) instead of creating another data_* TYPE_*
(new data type would probably have (data_string *) as base class)
(might revisit choice in the future)
HTTP_HEADER_UNSPECIFIED has been removed. It was used in select
locations as an optimization to avoid looking up enum header_header_e
before checking the array, but the ordering in the array now relies
on having the id. Having the id allows for a quick check if a common
header is present or not in the htags bitmask, before checking the
array, and allows for integer comparison in the log(n) search of the
array, instead of strncasecmp().
With HTTP_HEADER_UNSPECIFIED removed, add optimization to set bit
in htags for HTTP_HEADER_OTHER when an "other" header is added,
but do not clear the bit, as there might be addtl "other" headers
header field-names are required to be lowercase for HTTP/2
so specialize http_header_hkey_get() as http_header_hkey_get_lc()
for simpler comparison
lowercase field-names in http_headers[], as it does not matter for
buffer_eq_icase_ssn(), which is used with http_headers[] for HTTP/1.x
separate http_header_e index from r->{rqst,resp}_htags bitmask
(allows http_header_e remain 32-bit and to be used in array indexes
while also allowing r->{rqst,resp}_htags to grow to 64-bits wide
in the future, without requiring invasive changes)
allow h2.c layer to modify r->state in addition to r->h2state
Do not retire/release r on error if trailers (as opposed to headers)
(stream r was just allocated in the same scope for headers,
so ok to release)
log stream id with debug.log-state-handling in
connection_state_machine_loop()
(id will be 0 for HTTP/2 connection, but not for streams on connection)
(id will be 0 for HTTP/1.x requests)
(thx avij)
clients should send a single Cookie header with multiple cookie values
separated with ';'.
https://tools.ietf.org/html/rfc6265#section-4.2.1
However, HTTP/2 loosens this requirement for Cookie.
https://tools.ietf.org/html/rfc7540#section-8.1.2
Section 8.1.2.5 Compressing the Cookie Header Field
and some HTTP/2 clients (Chrome, Firefox) send multiple
'cookie:' headers in a HEADERS frame.
only XXH32() is used by ls-hpack, so disable XXH64() and XXH3()
to reduce code size. (This maybe be changed in the future if
XXH64() or XXH3() are utilitized for their performance)
(bug on master branch; never released)
(thx avij)
update doc that dir-listing.hide-dotfiles = "enable" by default
since lighttpd 1.4.40 https://redmine.lighttpd.net/issues/1081
(bug on master branch; never released)
(thx avij)
fix crash on master if blank line precedes HTTP/1.1 keep-alive request
header parsing code previously made assumptions that request was
HTTP/1.0 or HTTP/1.1, where a request-line was required, and which
would error out elsewhere if request-line was missing. The parsing
code also previously looked for "\r\n\r\n" to end headers.
The header offset parsing code was modified and invalidated the above
assumptions, now looking only for blank line "\r\n", but the calling
code had not properly been updated. (until this patch)
define LSHPACK_DEC_HTTP1X_OUTPUT 0
lighttpd does not require HTTP/1.1 output compat from HPACK decoder
("field-name: value\r\n")
define NDEBUG (in ls-hpack/lshpack.c)
lighttpd spends upwards of 20% total lighttpd CPU time in HPACK
encode/decode in h2load test on static file over cleartext (not TLS)
Defining NDEBUG eliminates some asserts() and results in a small
but measurable reduction in CPU usage
defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured
fork connection_get_state() for use by mod_status, which
might in the future choose to display different labels.
move and rename connection_get_short_state() into mod_status,
as the func is used only by mod_status
(connection_get_state() is currently used elsewhere only for debugging
in connections.c)
Empty field-name is already ignored when generating response headers,
but this is an explicit skip of empty field-name sent from backends,
including field-names that look like HTTP/2 pseudo-headers, as those
begin with ':'.
Location response header is permitted to use relative-path in
RFC 7231 Section 7.1.2. Location
Prefer relative path in redirection for the benefit of reverse proxies
and CDNs. Doing so also avoids potentially disclosing internal schemes
and server names which client might not be able to directly reach.
To restore prior behavior of sending a fully-qualified absolute URI:
server.feature-flags += ("absolute-dir-redirect" => "enable")
x-ref:
https://bz.apache.org/bugzilla/show_bug.cgi?id=63357
http_request_parse_header() specialized for HTTP/2 request headers
to be parsed as each field-name and value is HPACK-decoded; send headers
directly from HPACK decoder, rather than double-buffering in chunkqueue
http_request_headers_process_h2() for post-processing
LiteSpeed ls-hpack v2.2.1
XXX: might be better to include this as a git submodule
but minor code changes were made here for portability:
- C99 flexible array members defined as a[] instead of a[0])
- pedantic compiler warnings (excess ';' and missing declarations)
- deletion of large tables from ls-hpack/huff-tables.h (code size)
(subsequently incrementally updated using git rebase)
huge props and many thank yous to writers of testing tools used while
developing HTTP/2 support in lighttpd:
h2spec - conformance testing tool for HTTP/2 implementation
https://github.com/summerwind/h2spec
h2load - HTTP/2 benchmarking tool
https://nghttp2.org/documentation/h2load-howto.html
curl - command line tool and library for transferring data with URLs
https://curl.haxx.se/
move code from connections-glue.c back into connections.c
move code from connections-glue.c to http-header-glue.c
rename connection_response_reset()
to http_response_reset()
rename connection_handle_read_post_error()
to http_response_reqbody_read_error()
r->con->reqbody_read() replaces connection_handle_read_post_state()
future: might provide different callbacks for request body with
Content-Length versus request body sent via Transfer-Encoding: chunked
Note: rrdtool counts do not include HTTP/2 protocol overhead.
Continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
reset connection counters per connection, not per request
adjust mod_accesslog and mod_rrdtool usage
continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
Even though request headers are per-request, update the proto on the
connection level for trusted (proxy) clients. Note: the proxy must use
each connection only for a single connection from a single client
(typically true in practice), and not for multiple clients.
proto was previously saved on connection level for mod_extforward with
HAProxy PROXY protocol, but did not occur with X-Forwarded-Proto or the
Forwarded request header with proto=...
Before this change, modules which returned HANDLER_COMEBACK could lose
HTTPS=on if the resulting request was for a backend such as mod_fastcgi.
This was reported in mod_rewrite, but could also affect mod_magnet if
MAGNET_RESTART_REQUEST, or mod_cgi with cgi.local-redir = "enable"
x-ref:
"FastCGI behavior different when using rewrite?"
https://redmine.lighttpd.net/boards/2/topics/9293
gamin should be used instead of fam; fam is no longer maintained
This patch makes it safe to build lighttpd with gamin, but run on
systems with the (deprecated) fam installed, which can happen due
to historical package dependency declarations on some platforms
(Debian, Ubuntu).
gamin and fam are not 100% binary compatible.
(Among other things, fam does not provide FAMNoExists())
x-ref:
"llibgamin vs libfam conflict solving"
https://salsa.debian.org/debian/lighttpd/-/merge_requests/18
"libgamin0: libfam shlib dependency wrongly set to libfam0"
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510368
"undefined symbol: FAMNoExists"
https://bugs.launchpad.net/bugs/1453463
fix fallback if linkat() fails
check at startup if /proc/self/fd is present on systems with O_TMPFILE
(containers might not mount /proc)
x-ref:
"mod_webdav - PUT files with < 64kb Content-Length reults in zero length file"
https://redmine.lighttpd.net/boards/2/topics/9273
decode Transfer-Encoding: chunked from gw (gateway backends)
Transfer-Encoding: chunked is a hop-by-hop header.
Handling chunked encoding remove a hurdle for mod_proxy to send HTTP/1.1
requests to backends and be able to handle HTTP/1.1 responses.
Other backends ought not to send Transfer-Encoding: chunked, but in
practice, some implementations do.
Using BIO_new_mem_buf() allows lighttpd to control file reads and
securely wiping memory, avoiding the use of BIO_s_file(), which
employs stdio and BIO internal copying without securely wiping memory.
BIO_new_mem_buf() is also much more performant than BIO_s_file()
or BIO_new_fd() without a buffering BIO, which might make many
syscalls reading a single character at a time.
BIO_new_mem_buf() appears to be supported by older OpenSSL versions
as well as by numerous OpenSSL-compatible APIs, e.g. in BoringSSL,
LibreSSL, WolfSSL
translate config server.modules "mod_compress" to "mod_deflate"
accept compress.* directives, but issue DEPRECATED warning trace
mod_deflate differences from mod_compress:
- mod_compress compress.filetype was exact match; deflate.mimetypes is
prefix match (behavior change might compress longer mimetype matches,
which are likely of similar type and compressability)
- mod_compress always sent entire (compressed) file for Range request
mod_deflate will stream compress range result (not stored in cache)
- mod_compress would short-circuit request with 403 Forbidden error
if request file did not exist (stat() failed) (This behavior was
unfriendly to other handlers)
- mod_compress compress.cache-dir layout differs from deflate.cache-dir
layout; file cache should be cleared (or renamed) when migrating from
mod_compress to mod_deflate
- mod_deflate does not issue Vary: Accept-Encoding if request does not
contain Accept-Encoding. The identity response can be cache by
proxies and served to clients. Historically, some proxies disabled
caching if any Vary: response was seen. If the Vary header is
desirable, mod_deflate code which checks for Accept-Encoding and
compression type can be moved down a few lines to be below the
setting of the Vary response header.
use crypt() instead of crypt_r() to save stack space,
as struct crypt_data might be very large.
While crypt() is not thread-safe, lighttpd is single-threaded
auth.backend.ldap.timeout = "2000000" # quoted-string; microseconds
vhostdb.ldap += ("timeout" => "2000000") # quoted-string; microseconds
Default is 2000000 microseconds (2 secs)
These values are converted to struct timeval and passed to
ldap_set_option(ld, LDAP_OPT_NETWORK_TIMEOUT, );
ldap_set_option(ld, LDAP_OPT_TIMEOUT, ...);
if those LDAP_OPT_* values are available (both are OpenLDAP-specific).
x-ref:
"mod_auth caching"
https://redmine.lighttpd.net/issues/2805
auth.cache = ("max-age" => "600")
vhostdb.cache = ("max-age" => "600")
If specified with an empty array, default max-age is 600 secs (10 mins)
auth.cache = ()
vhostdb.cache = ()
(Note: cache expiration occurs every 8 seconds, so maximum cache time
might be up to max-age + 8 seconds)
x-ref:
"mod_auth caching"
https://redmine.lighttpd.net/issues/2805
Fixes the following error when building with -Dwith_maxminddb=true:
meson.build:916:1: ERROR: Unknown variable "libmaxminddb".
A full log can be found at meson-logs/meson-log.txt
Signed-off-by: Rosen Penev <rosenp@gmail.com>
libmariadb is what should be used as only the library portion is used.
Fixes compilation under OpenWrt.
Note that mariadb.pc is a superset that links to libmariadb.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
fix theoretical NULL dereference identified by Coverity Scan
possible for PROPFIND with specific atypical choices in lighttpd.conf:
- possible for getcontenttype if no content type matches resource
and no default type configured in lighttpd.conf
- possible for getetag if etag disabled in lighttpd.conf
(pedantic; no impact)
upon error, server will exit, so the impact of momentarily leaking fd
has no impact. This commit holds the fd in srv->stdin_fd to address
Coverity warning about leaking fd when using server.bind = "/dev/stdin"
add warning at server startup when mod_mysql_vhost is loaded
mod_vhostdb_mysql subsumes mod_mysql_vhost. Individual mod_mysql_vhost
directives map one-to-one to keywords in vhostdb.mysql = (...) directive
Glenn Strauss
Rosen Penev