short-circuit if response body completely received with response headers
x-ref:
"HTTP/2 requests sometimes take very long (missing last chunk)"
https://redmine.lighttpd.net/issues/3111
r->gw_dechunk->b is not a candidate for using generic chunk buffers.
chunked headers are generally smaller and fit in default 64 byte alloc.
Also, lighttpd limits chunked header to 1k.
Avoid unneeded optimization since HTTP/1.1 use is likely to diminish
over time in favor of HTTP/2 or HTTP/3 or later.
splice() data from backends to tempfiles (where splice() is available);
reduce copying data to userspace when writing data to tempfiles
Note: splice() on Linux returns EINVAL if target file has O_APPEND set
so lighttpd uses pwrite() (where available) when writing to tempfiles
(instead of lseek() + write(), or O_APPEND and write())
separate internal control for backend max_per_read
When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a smaller buffer to help reduce memory usage.
When not streaming, attempt to read and empty kernel socket bufs.
(e.g. MAX_READ_LIMIT 256k)
When writing to sockets (or pipes) attempt to fill kernel socket bufs.
(e.g. MAX_WRITE_LIMIT 256k)
reduce oversized memory allocations when reading from backends:
avoid extra power-2 allocation for 1 byte ('\0') when data
available to read is exactly power-2
return HANDLER_FINISHED from http_response_read() if response finished,
whether due to reading EOF (prior behavior), or if Content-Length was
provided and we have finished reading Content-Length, or if a module
sets r->resp_body_finished for any other reason. This may save an
unnecessary poll() and read() to receive EOF when Content-Length has
already been read.
manage r->resp_body_scratchpad in new funcs
http_response_append_buffer()
http_response_append_mem()
rather than
http_chunk_decode_append_buffer()
http_chunk_decode_append_mem()
which now only decode chunked encoding, more apropos for the func names
Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.
Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t
* Y2038: lighttpd 1.4.60 and later report Y2038 safety
$ lighttpd -V
+ Y2038 support # Y2038-SAFE
$ lighttpd -V
- Y2038 support (unsafe 32-bit signed time_t) # Y2038-UNSAFE
* Y2038: general platform info
* Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
- all major 64-bit platforms (known to this author) use 64-bit time_t
* Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
- Linux x32 ABI (different from i686)
- FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
- NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
- OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
- Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
- MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
* Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
- e.g. OpenVMS (unknown if lighttpd builds on this platform)
* Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
- Linux 32-bit (including i686)
- glibc 32-bit library support not yet available for 64-bit time_t
- https://sourceware.org/glibc/wiki/Y2038ProofnessDesign
- Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
https://itsubuntu.com/linux-kernel-5-6-to-fix-the-year-2038-issue-unix-y2k/
- https://www.gnu.org/software/libc/manual/html_node/64_002dbit-time-symbol-handling.html
"Note: at this point, 64-bit time support in dual-time
configurations is work-in-progress, so for these
configurations, the public API only makes the 32-bit time
support available. In a later change, the public API will
allow user code to choose the time size for a given
compilation unit."
- compiling with -D_TIME_BITS=64 currently has no effect
- glibc recent (Jul 2021) mailing list discussion
- https://public-inbox.org/bug-gnulib/878s2ozq70.fsf@oldenburg.str.redhat.com/T/
- FreeBSD i386
- DragonFlyBSD 32-bit
* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
* lighttpd prefers system monotonic clock instead of realtime clock
in places where realtime clock is not required
* lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
* (lighttpd presumes that lighttpd will not encounter dates before 1970
during normal operation.)
* lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
to convert negative timestamps for comparisions with 64-bit timestamps
(treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
* lighttpd provides unix_time64_t (int64_t) and
* lighttpd provides struct unix_timespec64 (unix_timespec64_t)
(struct timespec equivalent using unix_time64_t tv_sec member)
* lighttpd provides gmtime64_r() and localtime64_r() wrappers
for platforms 32-bit platforms using 32-bit time_t and
lighttpd temporarily shifts the year in order to use
gmtime_r() and localtime_r() (or gmtime() and localtime())
from standard libraries, before readjusting year and passing
struct tm to formatting functions such as strftime()
* lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
unsigned 32-bit and then to unix_time64_t
* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
on 32-bit platforms using 32-bit signed time_t, underlying libraries and
underlying filesystems might not behave properly after 32-bit signed time_t
overflows (19 Jan 2038 03:14:08 GMT). If a given 32-bit OS does not work
properly using negative time_t values, then lighttpd likely will not work
properly on that system.
* Other references and blogs
- https://en.wikipedia.org/wiki/Year_2038_problem
- https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs
- http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/
This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.
Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.
In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func. In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.
- check for empty strings at config time and set value to NULL if blank
string will be ignored at runtime; at runtime, simple pointer check
for NULL can be used to check for a value that has been set and is not
blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
and use buffer_is_unset() instead of buffer_is_empty(),
where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
truncate string, and use buffer_extend() to extend
Examples where buffer known not to be NULL:
- cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
(though we might set it to NULL if buffer_is_blank(cpv->v.b))
- address of buffer is arg (&foo)
(compiler optimizer detects this in most, but not all, cases)
- buffer is checked for NULL earlier in func
- buffer is accessed in same scope without a NULL check (e.g. b->ptr)
internal behavior change:
callers must not pass a NULL buffer to some funcs.
- buffer_init_buffer() requires non-null args
- buffer_copy_buffer() requires non-null args
- buffer_append_string_buffer() requires non-null args
- buffer_string_space() requires non-null arg
When lighttpd is not configured to stream the response body,
lighttpd sends partial content with an incorrect Content-Length
to an HTTP/1.0 client if a backend sends Transfer-Encoding: chunked
in response to lighttpd HTTP/1.1 request (to backend), and the response
from the backend ends up truncated.
lighttpd could instead send an HTTP/1.0 502 Bad Gateway, but the
current implementation chooses to send the partial content. After all,
an HTTP/1.0 client is, well, HTTP/1.0, and so of limited intelligence.
merge http_response_send_file 0-sized file special case
(historically was a short-circuit before Range handling,
but Range handling has been rewritten and moved elsewhere)
reuse cache lookup in common case of serving a static file
rather than repeating the stat_cache_entry lookup
(which is more work than memcmp() to re-check stat_cache_entry match)
RFC 7233 Range handling for all non-streaming responses,
including (non-streaming) dynamic responses
(previously Range responses handled only for static files)
(thx daimh)
commit 903024d7 in lighttpd 1.4.57 fixed issue #3046 but in the process
broke HTTP/1.1 101 Switching Protocols which included Content-Length: 0
in the response headers. Content-Length response header is permitted
by the RFCs, but not necessary with HTTP status 101 Switching Protocols.
x-ref:
"websocket proxy fails if 101 Switching Protocols from backend includes Content-Length"
https://redmine.lighttpd.net/issues/3063
(thx axe34)
Please note that this would not have crashed "x-sendfile-docroot"
were set to restrict the locations of files that can be sent via
X-Sendfile. If users are untrusted, "x-sendfile" should not be
enable without also configuring "x-sendfile-docroot"
x-ref:
"Server Aborted due to Malicious Data sent through CGI Sendfile"
https://redmine.lighttpd.net/issues/3062
(thx flynn)
fix large memory usage for large file downloads from dynamic backends
reuse or release large memory chunks
x-ref:
"Memory Growth with PUT and full buffered streams"
https://redmine.lighttpd.net/issues/3033
replace /* fall through */ comment with __attribute_fallthrough__ macro
Note: not adding attribute to code with external origins:
xxhash.h (algo_xxhash.h)
ls-hpack/lshpack.c
so to avoid warnings, may need to compile with -Wno-implicit-fallthrough
use http_chunk_append_file_ref() and http_chunk_append_file_ref_range()
reduce resource usage (number of fds open) by reference counting open
fds to files served, and sharing the fd among FILE_CHUNKs in responses
Glenn Strauss