create API in chunk.[ch] for writing a chunk to an fd
(pull similar code from mod_cgi and mod_webdav)
This new API is intended for use on request body input, which is
written to size-limited temporary files controlled by lighttpd and
written to files or pipes.
(network_backend_write() is for writing chunkqueues to sockets)
(thx flynn)
fix large memory usage for large file downloads from dynamic backends
reuse or release large memory chunks
x-ref:
"Memory Growth with PUT and full buffered streams"
https://redmine.lighttpd.net/issues/3033
(replace existing check which suffered from ToC-ToU race condition)
enhances logic from 2015 commit 593599f1 and avoids repeated fstat()
checks when sending large files
For mmap(), lighttpd catches SIGBUS if file is (externally) truncated
and lighttpd attempts to access bytes in a read-only mapping more than
a memory page boundary following the end of the file.
For sendfile(), lighttpd returns an error if sendfile() reports no error
and that no bytes have been sent after lighttpd attempts to send a
non-zero number of bytes.
defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured
convert all log_error_write() to log_error() and pass (log_error_st *)
use con->errh in preference to srv->errh (even though currently same)
avoid passing (server *) when previously used only for logging (errh)
code reuse, simplification, and inlining
remove excess calls to chunkqueue_remove_finished_chunks()
(it may still be possible for there to be an empty chunk in chunkqueue
if nothing were written to a temporary file (need to verify this), so
preserve some calls to chunkqueue_remove_finished_chunks() for now)
reduce initial buffer size if large POST to backend stored in temp files
regression in lighttpd 1.4.52
(thx rgenoud)
x-ref:
"[regression] lighttpd gets killed after uploading a big file"
https://redmine.lighttpd.net/issues/2922
quickly clear buffer instead of buffer_string_set_length(b, 0) or
buffer_reset(b). Avoids free() of large buffers about to be reused,
or buffers that are module-scoped, persistent, and reused.
(buffer_reset() should still be used with buffers in connection *con
when the data in the buffers is supplied by external, untrusted source)
server.chunkqueue-chunk-sz = 4096 (default)
Can be configured any integral value greater than 0.
Value is rounded up to next 1024 if not an even multiple of 1k.
Sites with large request or response headers may benefit from
an 8k or 16k chunk size. Embedded systems might choose to minimize
memory use by using a 1k chunk size.
Glenn Strauss