support Transfer-Encoding: chunked request body in conjunction with
server.stream-request-body = 0
dynamic handlers will still return 411 Length Required if
server.stream-request-body = 1 or 2 (!= 0)
since CGI-like env requires CONTENT_LENGTH be set
(and mod_proxy currently sends HTTP/1.0 requests to backends,
and Content-Length recommended for robust interaction with backend)
x-ref:
"request: support Chunked Transfer Coding for HTTP PUT"
https://redmine.lighttpd.net/issues/2156
(for persistence across multiple re-entries into routines upon
receiving fdevent)
(setting module debug flag in global scope is still recommended
since there are places where p->conf.debug is checked since
handler_ctx might not be available at all points)
Due to the POLLHUP behavior triggered on *BSD/Darwin, the shutdown()
had previously been limited to local connections. If interested in
squeezing the last bits of performance out of a machine, an admin
should configure local connections to be AF_UNIX instead of AF_INET
or AF_INET6 to localhost. The reason the shutdown() was originally
added in mod_proxy and mod_scgi was to aggressively reduce the number
of potential sockets in TIME_WAIT held by lighttpd.
(See commit:923688d2 "drain backend socket/pipe bufs upon FDEVENT_HUP",
done for reliability given the aforementioned *BSD/Darwin behavior.)
When using AF_UNIX, the TIME_WAIT issue does not exist, ergo, the
recommendation is to use AF_UNIX for local sockets, when available.
Using AF_UNIX sockets is a better solution to eliminate TIME_WAIT
than is TCP shutdown() half-close which, as we have seen, might not
be handled well by frameworks which are more complex than basic read
request, send response, and close.
x-ref:
"1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwin"
https://redmine.lighttpd.net/issues/2743
mod_fastcgi, mod_scgi, and mod_proxy must enforce wait for POLLWR
after EINPROGRESS or else getsockopt(fd, SOL_SOCKET, SO_ERROR, ...)
may succeed even though socket connection is not yet established,
and subsequent writev() will fail ENOTCONN.
(thx pkubaj)
x-ref:
"1.4.40/41 writev failed: Socket is not connected (fastcgi,scgi,proxy)"
https://redmine.lighttpd.net/issues/2744
shutdown(fd, SHUT_WR) after sending request to proxy or SCGI
only if remote is local and platform is not *BSD or Darwin.
The reason this fix is special-casing *BSD and Darwin is that the Single
Unix Specification and POSIX.1-2013 clearly specify that POLLHUP event
should be returned by poll only when the stream is no longer writable.
A half-closed socket that is still writable clearly does not match that
condition, yet that is what I am seeing on Darwin (El Capitan), and
presumably what others are seeing on *BSD, from which Apple originally
inherited the Darwin TCP stack.
Single Unix Specification (SUSv2) from 1997
(yes, that is nearly 20 years ago):
http://pubs.opengroup.org/onlinepubs/007908799/xsh/poll.html
POLLHUP
The device has been disconnected. This event and POLLOUT are
mutually exclusive; a stream can never be writable if a hangup has
occurred. However, this event and POLLIN, POLLRDNORM, POLLRDBAND or
POLLPRI are not mutually exclusive. This flag is only valid in the
revents bitmask; it is ignored in the events member.
Updated version of The Open Group Base Specifications Issue 7
(published in 2013):
http://pubs.opengroup.org/onlinepubs/9699919799/
POLLHUP
A device has been disconnected, or a pipe or FIFO has been closed
by the last process that had it open for writing. Once set, the
hangup state of a FIFO shall persist until some process opens the
FIFO for writing or until all read-only file descriptors for the
FIFO are closed. This event and POLLOUT are mutually-exclusive;
a stream can never be writable if a hangup has occurred. However,
this event and POLLIN, POLLRDNORM, POLLRDBAND, or POLLPRI are not
mutually-exclusive. This flag is only valid in the revents bitmask;
it shall be ignored in the events member.
x-ref:
"1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwin"
https://redmine.lighttpd.net/issues/2743
Strip bogus "Proxy" header before creating subprocess environment.
(mod_cgi, mod_fastcgi, mod_scgi, mod_ssi, mod_proxy)
Do not emit HTTP_PROXY to subprocess environment.
Some executables use HTTP_PROXY to configure outgoing proxy.
This is not a lighttpd security issue per se, but this change to
lighttpd adds a layer of defense to protect backend processes which
might be vulnerable due to blindly using this untrusted environment
variable. The HTTP_PROXY environment variable should not be trusted
by a program running in a CGI-like environment.
Mitigation in lighttpd <= 1.4.40 is to reject requests w/ Proxy header:
* Create "/path/to/deny-proxy.lua", read-only to lighttpd, with content:
if (lighty.request["Proxy"] == nil) then return 0 else return 403 end
* Modify lighttpd.conf to load mod_magnet and run lua code
server.modules += ( "mod_magnet" )
magnet.attract-raw-url-to = ( "/path/to/deny-proxy.lua" )
References:
https://www.kb.cert.org/vuls/id/797896
CGI web servers assign Proxy header values from client requests to
internal HTTP_PROXY environment variables
https://httpoxy.org/
httpoxy: A CGI application vulnerability
cygwin does not support ioctl on sockets, returning EOPTNOTSUPP
(would be better if cygwin used Windows ioctlsocket() instead)
Windows uses signed (socklen_t), so add some casts to quiet warnings
Windows path handling is convoluted, so disable one tests in mod_fastcgi
since trailing spaces are removed from URL for _WIN32 and __CYGWIN__ in
response.c
buffer.c:itostr() undefined behavior taking modulus of negative number
additional minor code changes made to quiet other coverity warnings
(false positives)
fd leak in mod_dirlisting.c
use after free in error condition in mod_proxy.c
NULL pointer dereference in error message in chunk.c
additional minor code changes made to quiet other coverity warnings
Set server.stream-response-body = 1 or server.stream-response-body = 2
to have lighttpd stream response body to client as it arrives from the
backend (CGI, FastCGI, SCGI, proxy).
default: buffer entire response body before sending response to client.
(This preserves existing behavior for now, but may in the future be
changed to stream response to client, which is the behavior more
commonly expected.)
x-ref:
"fastcgi, cgi, flush, php5 problem."
https://redmine.lighttpd.net/issues/949
"Random crashing on FreeBSD 6.1"
https://redmine.lighttpd.net/issues/760
"Memory usage increases when proxy+ssl+large file"
https://redmine.lighttpd.net/issues/1283
"lighttpd+fastcgi memory problem"
https://redmine.lighttpd.net/issues/1387
move code in dynamic handlers (CGI, FastCGI, SCGI, proxy)
from *_handle_fdevent() to *_recv_response() for reuse
outside the *_handle_fdevent() routine
Set server.stream-request-body = 1 or server.stream-request-body = 2
to have lighttpd connect to backend (CGI, FastCGI, SCGI, proxy)
immediately after parsing request headers, and to stream request body
as it arrives.
default: buffer entire request body before connecting to backend,
in order to avoid tying up (limited) backend resources which are often
implemented using libraries which wait for entire request body before
proceeding.
x-ref:
"Reimplement upload (POST) handling to match apache/zeus/thttpd/boa functionality"
https://redmine.lighttpd.net/issues/376
dynamic handlers mod_fastcgi, mod_scgi, and mod_proxy can now read
response from backend prior to finishing sending request body.
If the backend closes the connections (or shuts down socket write
end so that lighttpd read() 0 to indicate EOF), then lighttpd will
abort attempting to send request body to backend.
x-ref:
"mod_fastcgi should handle "quick" responses"
https://redmine.lighttpd.net/issues/2566
"FastCGI FCGI_STDOUT before FCGI_STDIN bug"
https://redmine.lighttpd.net/issues/131
to detect client disconnect. Do so even when waiting on backend,
and not polling for POLLRD or POLLWR on client connection.
This reduces unnecessary load on backends when backends are slow
to respond and client has given up waiting.
x-ref:
"https://redmine.lighttpd.net/issues/399"
FastCGI performance on high load
read request body in dynamic handlers supporting request body
(mod_cgi, mod_fastcgi, mod_proxy, mod_scgi, mod_webdav)
(In the future, each dynamic handler might choose whether or not to
buffer request body or to stream request body to backend as request
body is received.)
modify mod_webdav to mark request in handle_physical hook, and move
the main logic to handle_subrequest hook, where the main logic is
for other dynamic handlers.
read request body right before calling subrequest handler,
allowing request to be handled prior to reading request body,
e.g. to send 401 Unauthorized response when authentication is required
(In the future, this might move into each dynamic handler which supports
request body (mod_cgi, mod_fastcgi, mod_proxy, mod_scgi, mod_webdav) so
that each dynamic handler can choose whether or not to buffer request
body or to stream request body to backend as request body is received.)
keep-alive is disabled if request body has not been completely read
prior to sending response
x-ref:
"HTTP 401 Unauthorized only sent back after full POST request is read"
https://redmine.lighttpd.net/issues/2541
for dynamic handlers mod_cgi, mod_fastcgi, mod_scgi, mod_proxy
(mod_cgi control flow logic simplification began in a prior commit)
- connection state machine calls the subrequest handler
- subrequest handler sets up the connection to the backend
and registers fdevent handler to handle backend events
- fdevent handler handles backend events and then schedules
a call to connection state machine
- when retrying an alternate backend, backend state is reset
and then response state is reset so that the connection state
machine will call back into the subrequest handler to retry
Dynamic handlers no longer directly modify connection state
(calls to connection_set_state() from dynamic handlers were removed)
Dynamic handlers no longer reset con->physical.path, and they
preserve con->mode when retrying alternate backends. This is done
to skip repeated processing in response.c:http_response_prepare()
While this patch increases consistency in control flow handling,
there is more work to be done that can further improve upon this.
x-ref:
"handle-req time too long"
https://redmine.lighttpd.net/issues/1149
If the server is set to a path like value (starting with "/")
mod_proxy will try to establish a connection via unix domain socket.
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@3002 152afb58-edef-0310-8abb-c4023f1b3aa9
- a lot of code tried to handle manually adding terminating zeroes and
keeping track of the correct "used" count.
Replaced all "external" usages with simple wrapper functions:
* buffer_string_is_empty (used <= 1), buffer_is_empty (used == 0);
prefer buffer_string_is_empty
* buffer_string_set_length
* buffer_string_length
* CONST_BUF_LEN() macro
- removed "static" buffer hacks (buffers pointing to constant/stack
memory instead of malloc()ed data)
- buffer_append_strftime(): refactor buffer+strftime uses
- li_tohex(): no need for a buffer for binary-to-hex conversion:
the output data length is easy to predict
- remove "-Winline" from extra warnings: the "inline" keyword just
supresses the warning about unused but defined (static) functions;
don't care whether it actually gets inlined or not.
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2979 152afb58-edef-0310-8abb-c4023f1b3aa9
* removed almost all usages of buffer as "memory" (without terminating
zero)
* refactored cgi variable name encoding
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2977 152afb58-edef-0310-8abb-c4023f1b3aa9
Although those were "easy" to use, they violated the abstraction:
content of the chunkqueue should only be modified via the API.
Replace with chunkqueue_get_memory() and chunkqueue_use_memory() for
functions that read data from network (reusing large buffers),
chunkqueue_steal_with_tempfiles() to store request bodies on disk
temporarily.
Modules that were generating content and need a buffer maintain the
buffer manually (have to be careful to free the buffer on errors, as
it isn't part of the chunkqueue yet).
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2976 152afb58-edef-0310-8abb-c4023f1b3aa9
* remove unused structs and functions
(buffer_array, read_buffer)
* change return type from int to void for many functions,
as the return value (indicating error/success) was never checked,
and the function would only fail on programming errors and not on
invalid input; changed functions to use force_assert instead of
returning an error.
* all "len" parameters now are the real size of the memory to be read.
the length of strings is given always without the terminating 0.
* the "buffer" struct still counts the terminating 0 in ->used,
provide buffer_string_length() to get the length of a string in a
buffer.
unset config "strings" have used == 0, which is used in some places
to distinguish unset values from "" (empty string) values.
* most buffer usages should now use it as string container.
* optimise some buffer copying by "moving" data to other buffers
* use (u)intmax_t for generic int-to-string functions
* remove unused enum values: UNUSED_CHUNK, ENCODING_UNSET
* converted BUFFER_APPEND_SLASH to inline function (no macro feature
needed)
* refactor: create chunkqueue_steal: moving (partial) chunks into another
queue
* http_chunk: added separate function to terminate chunked body instead of
magic handling in http_chunk_append_mem().
http_chunk_append_* now handle empty chunks, and never terminate the
chunked body.
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2975 152afb58-edef-0310-8abb-c4023f1b3aa9
None of this matters - lighttpd will terminate anyway. Still helps the
code to get cleaner, and makes reviewing output of static analyzers
easier.
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2947 152afb58-edef-0310-8abb-c4023f1b3aa9
These should all be non critical:
* memory leaks on startup in error cases (which lead to
immediate shutdowns anyway)
* http_auth/ldap: passing uninitialized "ret" to ldap_err2string
* sizeof(T) not matching the target pointer in malloc/calloc calls;
those cases were either:
* T being the wrong pointer type - shouldn't matter as long as all
pointers have same size
* T being larger than the type needed
* mod_accesslog: direct use after free in cleanup (server shutdown);
could crash before "clean" shutdown
* some false positives (mod_compress, mod_expire)
* assert(srv->config_context->used > 0); - this is always the case,
as there is always a global config block
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2920 152afb58-edef-0310-8abb-c4023f1b3aa9
con->conf.is_ssl got removed and replaced by:
* con->conf.ssl_enabled for the config var "ssl.engine" - it is only
used to determine which server-sockets should use ssl. (usually not
needed as it is mandatory and enough to set ssl.pemfile anyway)
* con->srv_socket->is_ssl to detect the actual ssl status of the
bound socket, which is the same as the ssl status of the connection
* con->uri.scheme for the actual $HTTP["scheme"] value, also used for
the CGI "HTTPS=ON" variable. This defaults to "https" if the
connection uses ssl, but can be changed for example by mod_extforward
if X-Forwarded-Proto: is set to either "http" or "https" (other values
are ignored right now)
Also removed the broken srv_socket->is_proxy_ssl as it was a connection
value in a server_socket struct...
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2887 152afb58-edef-0310-8abb-c4023f1b3aa9
There are 6 remaining occurrences after this commit, in mod_compress,
due to zlib's Z_NULL being defined as 0 instead of NULL.
Signed-off-by: Cyril Brulebois <kibi@debian.org>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2845 152afb58-edef-0310-8abb-c4023f1b3aa9
The proper way to declare a function taking no parameters isn't:
foo bar();
But this instead:
foo bar(void);
Signed-off-by: Cyril Brulebois <kibi@debian.org>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2843 152afb58-edef-0310-8abb-c4023f1b3aa9
Pascal Bach