add comment for handling of streaming with no response body
add commented-out code to disable streaming to wait for backend protocol
to signal end of response (prevent http_response_write_prepare() from
short-circuiting and finishing responses without response body)
x-ref:
"FastCGI premature socket close with response streaming and 204 status"
https://redmine.lighttpd.net/boards/2/topics/10066
disable streaming response while processing "authorizer" mode
until "authorizer" response 200 OK from the backend is complete
(thx jefftharris)
x-ref:
"FastCGI authorizer hang with server.stream-response-body"
https://redmine.lighttpd.net/boards/2/topics/9969
"FastCGI authorizer hang with server.stream-response-body"
https://redmine.lighttpd.net/issues/3106
separate internal control for backend max_per_read
When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a smaller buffer to help reduce memory usage.
When not streaming, attempt to read and empty kernel socket bufs.
(e.g. MAX_READ_LIMIT 256k)
When writing to sockets (or pipes) attempt to fill kernel socket bufs.
(e.g. MAX_WRITE_LIMIT 256k)
rename log_error_multiline_buffer() to log_error_multiline()
and take (char *)ptr and (size_t)len instead of (buffer *)b
When debug printing request and response headers,
print each header on separate line for readability
and omit '\r' if "\r\n" ends line
This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.
Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.
In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func. In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.
- check for empty strings at config time and set value to NULL if blank
string will be ignored at runtime; at runtime, simple pointer check
for NULL can be used to check for a value that has been set and is not
blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
and use buffer_is_unset() instead of buffer_is_empty(),
where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
truncate string, and use buffer_extend() to extend
Examples where buffer known not to be NULL:
- cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
(though we might set it to NULL if buffer_is_blank(cpv->v.b))
- address of buffer is arg (&foo)
(compiler optimizer detects this in most, but not all, cases)
- buffer is checked for NULL earlier in func
- buffer is accessed in same scope without a NULL check (e.g. b->ptr)
internal behavior change:
callers must not pass a NULL buffer to some funcs.
- buffer_init_buffer() requires non-null args
- buffer_copy_buffer() requires non-null args
- buffer_append_string_buffer() requires non-null args
- buffer_string_space() requires non-null arg
optimize buffer_* primitives
Other than buffer_string_set_length(), reallocate with one power-2 step
in size (or use the requested size, if larger). This replaces the fixed
BUFFER_PIECE_SIZE round-up of only 64 bytes extension each reallocation,
which could lead to excessive reallocations in some scenarios.
buffer_extend() convenience routine to prep for batch append
(combines buffer_string_prepare_append() and buffer_commit())
mod_fastcgi, mod_scgi, mod_proxy and others now leverage buffer_extend()
mod_scgi directly performs little-endian encoding of short ints
http_response_write_header() optimizes writing response header,
leveraging buffer_extend()
modify mod_proxy to append line ends
similar to how it is done in http_response_write_header()
(removes one call to buffer_append_string_len())
- rewrite fcgi-responder as standalone app
fcgi-responder is now a minimal, standalone FastCGI server for tests
- remove dependency on fcgi-devel package
- merge fcgi-auth into fcgi-responder
When server.stream-request-body = 0 (the default), the entire request
body is collected before engaging the backend. For backends which
require data framing, this could lead to growth in memory use as large
requests were framed all at once.
Prefer to retain large request bodies in temporary files on disk and
frame in portions as write queue to backend drains below a threshold.
x-ref:
"Memory Growth with PUT and full buffered streams"
https://redmine.lighttpd.net/issues/3033
(expansion of buffer_string_lenth() inline function and CONST_BUF_LEN()
macro, which always check for NULL, appears to cause the analyzer to
believe that a pointer might be NULL in cases where it otherwise can
not be NULL)
x-ref:
http://clang-analyzer.llvm.org/faq.html
NB: r->tmp_buf == srv->tmp_buf (pointer is copied for quicker access)
NB: request read and write chunkqueues currently point to connection
chunkqueues; per-request and per-connection chunkqueues are
not distinct from one another
con->read_queue == r->read_queue
con->write_queue == r->write_queue
NB: in the future, a separate connection config may be needed for
connection-level module hooks. Similarly, might need to have
per-request chunkqueues separate from per-connection chunkqueues.
Should probably also have a request_reset() which is distinct from
connection_reset().
use global rather than passing around (server *) just for that
li_itostrn() and li_utostrn() return string length
(rather than requiring subsequent strlen() to find length)
convert all log_error_write() to log_error() and pass (log_error_st *)
use con->errh in preference to srv->errh (even though currently same)
avoid passing (server *) when previously used only for logging (errh)
array_get_element_klen() is now intended for read-only access
array_get_data_unset() is used by config processing for r/w access
array_get_buf_ptr() is used for r/w access to ds->value (string buffer)
reduce initial buffer size if large POST to backend stored in temp files
regression in lighttpd 1.4.52
(thx rgenoud)
x-ref:
"[regression] lighttpd gets killed after uploading a big file"
https://redmine.lighttpd.net/issues/2922
quickly clear buffer instead of buffer_string_set_length(b, 0) or
buffer_reset(b). Avoids free() of large buffers about to be reused,
or buffers that are module-scoped, persistent, and reused.
(buffer_reset() should still be used with buffers in connection *con
when the data in the buffers is supplied by external, untrusted source)
Glenn Strauss
Mohammed Sadiq