disable streaming response while processing "authorizer" mode
until "authorizer" response 200 OK from the backend is complete
(thx jefftharris)
x-ref:
"FastCGI authorizer hang with server.stream-response-body"
https://redmine.lighttpd.net/boards/2/topics/9969
"FastCGI authorizer hang with server.stream-response-body"
https://redmine.lighttpd.net/issues/3106
separate internal control for backend max_per_read
When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a smaller buffer to help reduce memory usage.
When not streaming, attempt to read and empty kernel socket bufs.
(e.g. MAX_READ_LIMIT 256k)
When writing to sockets (or pipes) attempt to fill kernel socket bufs.
(e.g. MAX_WRITE_LIMIT 256k)
Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.
Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t
* Y2038: lighttpd 1.4.60 and later report Y2038 safety
$ lighttpd -V
+ Y2038 support # Y2038-SAFE
$ lighttpd -V
- Y2038 support (unsafe 32-bit signed time_t) # Y2038-UNSAFE
* Y2038: general platform info
* Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
- all major 64-bit platforms (known to this author) use 64-bit time_t
* Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
- Linux x32 ABI (different from i686)
- FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
- NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
- OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
- Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
- MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
* Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
- e.g. OpenVMS (unknown if lighttpd builds on this platform)
* Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
- Linux 32-bit (including i686)
- glibc 32-bit library support not yet available for 64-bit time_t
- https://sourceware.org/glibc/wiki/Y2038ProofnessDesign
- Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
https://itsubuntu.com/linux-kernel-5-6-to-fix-the-year-2038-issue-unix-y2k/
- https://www.gnu.org/software/libc/manual/html_node/64_002dbit-time-symbol-handling.html
"Note: at this point, 64-bit time support in dual-time
configurations is work-in-progress, so for these
configurations, the public API only makes the 32-bit time
support available. In a later change, the public API will
allow user code to choose the time size for a given
compilation unit."
- compiling with -D_TIME_BITS=64 currently has no effect
- glibc recent (Jul 2021) mailing list discussion
- https://public-inbox.org/bug-gnulib/878s2ozq70.fsf@oldenburg.str.redhat.com/T/
- FreeBSD i386
- DragonFlyBSD 32-bit
* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
* lighttpd prefers system monotonic clock instead of realtime clock
in places where realtime clock is not required
* lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
* (lighttpd presumes that lighttpd will not encounter dates before 1970
during normal operation.)
* lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
to convert negative timestamps for comparisions with 64-bit timestamps
(treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
* lighttpd provides unix_time64_t (int64_t) and
* lighttpd provides struct unix_timespec64 (unix_timespec64_t)
(struct timespec equivalent using unix_time64_t tv_sec member)
* lighttpd provides gmtime64_r() and localtime64_r() wrappers
for platforms 32-bit platforms using 32-bit time_t and
lighttpd temporarily shifts the year in order to use
gmtime_r() and localtime_r() (or gmtime() and localtime())
from standard libraries, before readjusting year and passing
struct tm to formatting functions such as strftime()
* lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
unsigned 32-bit and then to unix_time64_t
* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
on 32-bit platforms using 32-bit signed time_t, underlying libraries and
underlying filesystems might not behave properly after 32-bit signed time_t
overflows (19 Jan 2038 03:14:08 GMT). If a given 32-bit OS does not work
properly using negative time_t values, then lighttpd likely will not work
properly on that system.
* Other references and blogs
- https://en.wikipedia.org/wiki/Year_2038_problem
- https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs
- http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/
reuse cache lookup in common case of serving a static file
rather than repeating the stat_cache_entry lookup
(which is more work than memcmp() to re-check stat_cache_entry match)
relay 1xx from backend over HTTP/2, e.g. 103 Early Hints
(if client is connected using HTTP/2)
enabled by default unless disabled in lighttpd.conf with:
server.feature-flags += ( "server.h2-discard-backend-1xx" = "enable" )
Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response. Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request.
https://tools.ietf.org/html/rfc8297#section-3 Security Considerations
x-ref:
An HTTP Status Code for Indicating Hints (103 Early Hints)
https://tools.ietf.org/html/rfc8297
move code from connections-glue.c back into connections.c
move code from connections-glue.c to http-header-glue.c
rename connection_response_reset()
to http_response_reset()
rename connection_handle_read_post_error()
to http_response_reqbody_read_error()
group HANDLER_COMEBACK logic in http_response_comeback() and call it
from places that reset state in order to (sometimes partially) reprocess
a request. This includes error handler (server.error-handler),
r->handler_module when cgi.local-redir, and looping in
http_response_prepare() when modules make changes to the request and
return HANDLER_COMEBACK (e.g. mod_rewrite, mod_magnet, mod_cml)
Also, set r->conditional_is_valid closer to where elements are set
(and become valid for use in condition checks), and parse target
in http_request_parse() instead of http_response_prepare()
NB: r->tmp_buf == srv->tmp_buf (pointer is copied for quicker access)
NB: request read and write chunkqueues currently point to connection
chunkqueues; per-request and per-connection chunkqueues are
not distinct from one another
con->read_queue == r->read_queue
con->write_queue == r->write_queue
NB: in the future, a separate connection config may be needed for
connection-level module hooks. Similarly, might need to have
per-request chunkqueues separate from per-connection chunkqueues.
Should probably also have a request_reset() which is distinct from
connection_reset().
convert all log_error_write() to log_error() and pass (log_error_st *)
use con->errh in preference to srv->errh (even though currently same)
avoid passing (server *) when previously used only for logging (errh)
provide standard types in first.h instead of base.h
provide lighttpd types in base_decls.h instead of settings.h
reduce headers exposed by headers for core data structures
do not expose <pcre.h> or <stdlib.h> in headers
move stat_cache_entry to stat_cache.h
reduce use of "server.h" and "base.h" in headers
common codebase for socket backends, based off mod_fastcgi with
some features added for mod_proxy
(mostly intended to reduce code duplication and enhance code isolation)
mod_fastcgi and mod_scgi can now use fastcgi.balance and scgi.balance
for similar behavior as proxy.balance, but the balancing is per-host
and not per-proc. proxy.balance is also per-host and not per-proc.
mod_proxy and mod_scgi can now use proxy.map-extensions and
scgi.map-extensions, similar to fastcgi.map-extensions.
mod_fastcgi behavior change (affects only mod_status):
- statistics tags have been renamed from "fastcgi.*" to "gw.*"
"fastcgi.backend.*" -> "gw.backend.*"
"fastcgi.active-requests" -> "gw.active-requests"
("fastcgi.requests" remains "fastcgi.requests")
("proxy.requests" is new)
("scgi.requests" is new)
mod_scgi behavior change (likely minor):
- removed scgi_proclist_sort_down() and scgi_proclist_sort_up().
procs now chosen based on load as measured by num socket connnections
Note:
modules using gw_backend.[ch] are currently still independent modules.
If it had been written as a single module with fastcgi, scgi, proxy
implementations, then there would have been a chance of breaking some
existing user configurations where module ordering made a difference
for which module handled a given request, though for most people, this
would have made no difference.
Details about mod_fastcgi code transformations:
unsigned int debug -> int debug
fastcgi_env member removed from plugin_config
renamed "fcgi" and "fastcgi" to "gw", and "FCGI" to "GW"
reorganize routines for high-level and lower-level interfaces
some lower-level internal interfaces changed to use host,proc,debug
args rather than knowing about higher-level (app) hctx and plugin_data
tabs->spaces and reformatting
Use same funcs as other dynamic handlers to recv data from backend.
Add hook for fastcgi to process FastCGI packets (and other future
dynamic handlers may hook this in order to handle custom data framing)
large code move, but minimal changes made to code (besides whitespace),
so that code builds
next: need to isolate openssl data structures and config parsing
initialized for mod_magnet and dynamic CGI-like handlers
(mod_cgi, mod_fastcgi, mod_scgi, mod_ssi) (*not* mod_proxy)
Note: in the future a config flag (does not yet exist) might be required
to activate initialization of these SSL_* env variables. This might
occur if there are requests to access these variables in mod_accesslog,
and/or if more SSL_* varables are created, which would be more work.
x-ref:
"pass protocol and cipher details to fcgi env"
https://redmine.lighttpd.net/issues/2511
consolidated from CGI, FastCGI, SCGI, SSI
Note: due to prior inconsistencies between the code in mod_cgi,
mod_fastcgi, mod_scgi, and mod_ssi, there are some minor behavior
changes.
CONTENT_LENGTH is now always set, even if 0
(though CONTENT_LENGTH is never set for FASTCGI_AUTHORIZER)
PATH_INFO is created only if present, not if empty.
(mod_fastcgi and mod_ssi previously set PATH_INFO="" (blank value))
PATH_TRANSLATED is now set if PATH_INFO is present
(previously missing from mod_cgi and mod_ssi)
mod_ssi now sets DOCUMENT_ROOT to con->physical.basedir, like others
(previously, mod_ssi set DOCUMENT_ROOT to con->physical.doc_root,
which matched con->physical.basedir unless mod_alias changed basedir)
mod_ssi now sets REQUEST_URI to con->request.orig_uri, like others
(previously, mod_ssi set REQUEST_URI to con->request.uri, which
matched con->request.orig_uri except after redirects, error docs)
handle X-Sendfile and X-LIGHTTPD-send-file w/ http_response_xsendfile()
if host is configured ( "x-sendfile" = "enable" )
Note: X-Sendfile path is url-decoded for consistency, like X-Sendfile2
(response headers should be url-encoded to avoid tripping over
chars allowed in filesystem but which might change response
header parsing semantics)
Note: deprecated: "allow-x-send-file"; use "x-sendfile"
Note: deprecated: X-LIGHTTPD-send-file header; use X-Sendfile header
Note: deprecated: X-Sendfile2 header; use X-Sendfile header
For now, X-Sendfile2 is still handled internally by mod_fastcgi.
Since http_response_send_file() supports HTTP Range requests,
X-Sendfile2 is effectively obsolete. However, any code, e.g. PHP,
currently using X-Sendfile2 is probably manually generating 206 Partial
Content status and Range response headers. A future version of lighttpd
might *remove* X-Sendfile2. Existing code should be converted to use
X-Sendfile, which is easily done by removing all the special logic
around using X-Sendfile2, since the 206 Partial Content status and Range
response headers are handled in http_response_send_file().
x-ref:
"mod_fastcgi + X-Sendfile -> mod_staticfile"
https://redmine.lighttpd.net/issues/799
"Feature Request: New option "x-send-file-docroot""
https://redmine.lighttpd.net/issues/851
"X-Sendfile handoff to mod-static-file in 1.4.x"
https://redmine.lighttpd.net/issues/2017
"X-sendfile should be able to set content-type"
https://redmine.lighttpd.net/issues/2076
move code from mod_staticfile.c to http-header-glue.c to allow reuse
(includes ETag, Last-Modified headers, Range requests for static files)
operate on path arg instead of con->physical.path
skip Range requests if con->http_status already set >= 300
remove redundant calls to stat_cache_get_entry() handling Range requests
x-ref:
"X-Sendfile handoff to mod-static-file in 1.4.x"
https://redmine.lighttpd.net/issues/2017
Glenn Strauss
Stefan Bühler
Elan Ruusamäe
Jan Kneschke