Omit calling cgi_handle_fdevent() after CGI process exit.
Another (sub)process may be holding pipe fd open and might write
response instead of the initial CGI process.
add FDEVENT_IN in addition to FDEVENT_HUP when triggering
cgi_handle_fdevent() after the CGI process exits.
(This helps improve reliability when running tests under Cygwin)
centralize most waitpid() handling in core server, with hooks for
modules to be informed of pid and status when a process exits.
This enables faster discovery (and restart) of exited processes,
and also allows for lighttpd to manage backend processes in the
parent (master) process when server.max-worker > 0.
common codebase for socket backends, based off mod_fastcgi with
some features added for mod_proxy
(mostly intended to reduce code duplication and enhance code isolation)
mod_fastcgi and mod_scgi can now use fastcgi.balance and scgi.balance
for similar behavior as proxy.balance, but the balancing is per-host
and not per-proc. proxy.balance is also per-host and not per-proc.
mod_proxy and mod_scgi can now use proxy.map-extensions and
scgi.map-extensions, similar to fastcgi.map-extensions.
mod_fastcgi behavior change (affects only mod_status):
- statistics tags have been renamed from "fastcgi.*" to "gw.*"
"fastcgi.backend.*" -> "gw.backend.*"
"fastcgi.active-requests" -> "gw.active-requests"
("fastcgi.requests" remains "fastcgi.requests")
("proxy.requests" is new)
("scgi.requests" is new)
mod_scgi behavior change (likely minor):
- removed scgi_proclist_sort_down() and scgi_proclist_sort_up().
procs now chosen based on load as measured by num socket connnections
Note:
modules using gw_backend.[ch] are currently still independent modules.
If it had been written as a single module with fastcgi, scgi, proxy
implementations, then there would have been a chance of breaking some
existing user configurations where module ordering made a difference
for which module handled a given request, though for most people, this
would have made no difference.
Details about mod_fastcgi code transformations:
unsigned int debug -> int debug
fastcgi_env member removed from plugin_config
renamed "fcgi" and "fastcgi" to "gw", and "FCGI" to "GW"
reorganize routines for high-level and lower-level interfaces
some lower-level internal interfaces changed to use host,proc,debug
args rather than knowing about higher-level (app) hctx and plugin_data
tabs->spaces and reformatting
fix streaming response when server.stream-response-body = 2
and client catches up to stream from backend
(thx horgh)
x-ref:
"mod_fastcgi can fail to read entire response from server"
https://redmine.lighttpd.net/issues/2796
More specific checks on contents of array lists. Each module using
lists now does better checking on the types of values in the list
(strings, integers, arrays/lists)
This helps prevent misconfiguration of things like cgi.assign,
fastcgi.server, and scgi.server, where source code might be
served as static files if parenthesis are misplaced.
x-ref:
https://redmine.lighttpd.net/boards/2/topics/6571
new directive cgi.local-redir = [enable|disable]
*disable* RFC3875 6.2.2 local-redir by default.
(behavior change from when local-redir support added in lighttpd 1.4.40)
The reason for this behavior change is that CGI local-redir support
(RFC3875 6.2.2) is an optimization. Absence of support may result in
additional latency in servicing a request due the additional round-trip
to the client, but that was the prior behavior (before lighttpd 1.4.40)
and is the behavior of web servers which do not support CGI local-redir.
However, enabling CGI local-redir by default may result in broken links
in the case where a user config (unaware of CGI local-redir behavior)
returns HTML pages containing *relative* paths (not root-relative paths)
which are relative to the location of the local-redir target document,
and the local-redir target document is located at a different URL-path
from the original CGI request.
x-ref:
RFC3875 CGI 1.1 specification section 6.2.2 Local Redirect Response
http://www.ietf.org/rfc/rfc3875
"CGI local redirect not implemented correctly"
https://redmine.lighttpd.net/issues/2108
"1.4.40 regression: broken redirect (using Location) between url.rewrite-once URLs"
https://redmine.lighttpd.net/issues/2793
set status 200 OK if CGI does not return CGI headers
Note:
This mode in lighttpd is deprecated and may be removed in the next major
release of lighttpd. CGI scripts should return a proper CGI header in
the response, even if that header is empty and followed by a blank line,
before return response body.
Without a proper CGI response header, the first line(s) of the response
might be incorrectly construed as being CGI response headers, especially
if they contain ':', and response may be corrupted. That is why this
mode is deprecated (and not supported in numerous other web servers).
The minimal valid CGI response header is "\n", which lighttpd will treat
as equivalent to "Status: 200\n\n"
x-ref:
"error 500 (mod_cgi.c.601) cgi died"
https://redmine.lighttpd.net/issues/2786
It is still not a good idea for backend to send Transfer-Encoding unless
backend is mod_proxy, and mod_proxy should not currently receive chunked
response since mod_proxy sends HTTP/1.0 request.
If mod_proxy is changed to sent HTTP/1.1 request, then lighttpd would
need to check if client is HTTP/1.0 and would need to de-chunk and
remove any other transfer-codings if not supported by next-hop.
x-ref:
"error 500 (mod_cgi.c.601) cgi died"
https://redmine.lighttpd.net/issues/2786
"Status" from CGI/1.1 environment should not be sent back to client.
Also, do not send "Status" back to client in mod_scgi
and more precisely parse for "Status" in mod_fastcgi
RFC3875 CGI local-redir stricter adherence
do not apply local-redir if any response headers besides "Location"
do not apply local-redir if any response body has been received
(though it might not have been received yet, and we do not wait to find
out, if lighttpd is configured to stream response body back to client)
x-ref:
RFC3875 CGI 1.1 specification section 6.2.2 Local Redirect Response
http://www.ietf.org/rfc/rfc3875
"CGI local redirect not implemented correctly"
https://redmine.lighttpd.net/issues/2108
Loosen local redirect handling in mod_cgi to skip handling as local
redirect if the Location matches con->uri.path, since if the request
is intended to redirect back to the same CGI using the same request
method, path info, and query string, the CGI would logically just
return the final intended response. Loosening this handling avoids a
problem with applications (potentially) accessible through multiple
gateways, where the application is not aware of this specific handling
of Location in the Common Gateway Interface (CGI/1.1), the application
sends abs-path in the Location response header instead of absoluteURI,
and the application expects the client to receive this Location response
header instead of the server to process as a CGI local redirect.
One example of such an application is LuCI,
which sends Set-Cookie with Location: /abs-path
https://github.com/openwrt/luci
(Note that this loose check for matching con->uri.path is not perfect
and might not match if the CGI returned a path with a different case
and the server is on a case-insensitive filesystem, or if the path
returned by the CGI is rewritten elsewhere to a different con->uri.path
before getting to mod_cgi.)
RFC3875 CGI 1.1 specification section 6.2.2 Local Redirect Response
http://www.ietf.org/rfc/rfc3875
x-ref:
"CGI local-redir handling conflicts with LuCI redirect w/ Set-Cookie"
https://redmine.lighttpd.net/issues/2779
"CGI local redirect not implemented correctly"
https://redmine.lighttpd.net/issues/2108
This change should fix an issue with lighttpd on Debian kfreebsd-* arch
(kfreebsd-amd64 and kfreebsd-i386)
.libs/mod_cgi.o: In function `cgi_create_env':
./src/mod_cgi.c:1103: warning: pipe2 is not implemented and will always fail
lighttpd is single-threaded so there is no race with pipe()
and then fcntl() F_SETFD FD_CLOEXEC on the pair of pipe fds.
Using pipe2() where available is still slightly more efficient
by eliding the syscalls to set FD_CLOEXEC.
Lack of pipe2() on relic Unix as well as missing on Mac OSX is likely
one reason why threaded web servers such as nginx choose not to support
CGI except via an external service to the process. Without pipe2(),
race conditions exist and it is not safe for a threaded server to use
pipe() and fork() when the server also does not want to potentially leak
open file descriptors to various unrelated CGI scripts.
support Transfer-Encoding: chunked request body in conjunction with
server.stream-request-body = 0
dynamic handlers will still return 411 Length Required if
server.stream-request-body = 1 or 2 (!= 0)
since CGI-like env requires CONTENT_LENGTH be set
(and mod_proxy currently sends HTTP/1.0 requests to backends,
and Content-Length recommended for robust interaction with backend)
x-ref:
"request: support Chunked Transfer Coding for HTTP PUT"
https://redmine.lighttpd.net/issues/2156
FreeBSD 9.3 has O_CLOEXEC, but does not have pipe2() until FreeBSD 10.
FreeBSD 10 also adds SOCK_CLOEXEC, so use that as indicator
https://wiki.freebsd.org/AtomicCloseOnExec
Note: FreeBSD 9.3 will soon reach its "extended" EOL date (31 Dec 2016),
so those using FreeBSD 9.3 should consider upgrading.
https://www.freebsd.org/security/security.html#sup
Mac OS X does not have pipe2(). User reported:
Undefined symbols for architecture x86_64:
"_pipe2", referenced from:
_mod_cgi_handle_subrequest in mod_cgi.o
x-ref:
"Undefined symbols _pipe2"
https://redmine.lighttpd.net/issues/2765
consolidated from CGI, FastCGI, SCGI, SSI
Note: due to prior inconsistencies between the code in mod_cgi,
mod_fastcgi, mod_scgi, and mod_ssi, there are some minor behavior
changes.
CONTENT_LENGTH is now always set, even if 0
(though CONTENT_LENGTH is never set for FASTCGI_AUTHORIZER)
PATH_INFO is created only if present, not if empty.
(mod_fastcgi and mod_ssi previously set PATH_INFO="" (blank value))
PATH_TRANSLATED is now set if PATH_INFO is present
(previously missing from mod_cgi and mod_ssi)
mod_ssi now sets DOCUMENT_ROOT to con->physical.basedir, like others
(previously, mod_ssi set DOCUMENT_ROOT to con->physical.doc_root,
which matched con->physical.basedir unless mod_alias changed basedir)
mod_ssi now sets REQUEST_URI to con->request.orig_uri, like others
(previously, mod_ssi set REQUEST_URI to con->request.uri, which
matched con->request.orig_uri except after redirects, error docs)
reverts part of commit:dbdab5db which swapped REQUEST_URI, REDIRECT_URI
x-ref:
"mediawiki redirect loop if REQUEST_URI not orig req in 1.4.40"
https://redmine.lighttpd.net/issues/2738
Explanation:
REQUEST_URI and REDIRECT_URI are not part of CGI standard environment.
The reason for their existence is that PATH_INFO in CGI environment may
be different from the path in the current request. The main reason for
this potential difference is that the URI path is normalized to a path
in the filesystem and tested against the filesystem to determine which
part is SCRIPT_NAME and which part is PATH_INFO. In case-insensitive
filesystems, the URI might be lowercased before testing against the
filesystem, leading to loss of case-sensitive submission in any
resulting PATH_INFO. Also, duplicated slashes "///" and directory
references "/." and "/.." are removed, including prior path component in
the case of "/..". This might be undesirable when the information after
the SCRIPT_NAME is virtual information and there target script needs the
virtual path preserved as-is. In that case, the target script can
re-parse REQUEST_URI (or REDIRECT_URI, as appropriate) to obtain the
unmodified information from the URI.
con->request.uri is equivalent to con->request.orig_uri unless the
request has been internally rewritten (e.g. by mod_rewrite, mod_magnet,
others), in which case con->request.orig_uri is the request made by the
client, and con->request.uri is the current URI being processed.
Historical REQUEST_URI (environment variable) lighttpd inconsistencies
- mod_cml set REQUEST_URI to con->request.orig_uri
- mod_cgi set REQUEST_URI to con->request.orig_uri
- mod_fastcgi set REQUEST_URI to con->request.orig_uri
- mod_scgi set REQUEST_URI to con->request.orig_uri
- mod_ssi set REQUEST_URI to current con->request.uri
- mod_magnet set MAGNET_ENV_REQUEST_URI to current con->request.uri
and MAGNET_ENV_REQUEST_ORIG_URI to con->request.orig_uri
Historical REDIRECT_URI (environment variable) previously set only in
mod_fastcgi and mod_scgi, and set to con->request.uri
Since lighttpd 1.4.40 provides REDIRECT_URI with con->request.orig_uri,
changes were made to REQUEST_URI for consistency, with the hope that
there would be little impact to existing configurations since the
request uri and original request uri are the same unless there has been
an internal redirect. It turns out that various PHP frameworks use
REQUEST_URI and require that it be the original URI requested by client.
Therefore, this change is being reverted, and lighttpd will set
REQUEST_URI to con->request.orig_uri in mod_cgi, mod_fastcgi, mod_scgi
as was done in lighttpd 1.4.39 and earlier. Similarly, REDIRECT_URI
also has the prior behavior in mod_fastcgi and mod_scgi, and added to
mod_cgi.
A future release of lighttpd might change mod_ssi to be consistent with
the other modules in setting REQUEST_URI to con->request.orig_uri and to
add REDIRECT_URI, when an internal redirect has occurred.
Strip bogus "Proxy" header before creating subprocess environment.
(mod_cgi, mod_fastcgi, mod_scgi, mod_ssi, mod_proxy)
Do not emit HTTP_PROXY to subprocess environment.
Some executables use HTTP_PROXY to configure outgoing proxy.
This is not a lighttpd security issue per se, but this change to
lighttpd adds a layer of defense to protect backend processes which
might be vulnerable due to blindly using this untrusted environment
variable. The HTTP_PROXY environment variable should not be trusted
by a program running in a CGI-like environment.
Mitigation in lighttpd <= 1.4.40 is to reject requests w/ Proxy header:
* Create "/path/to/deny-proxy.lua", read-only to lighttpd, with content:
if (lighty.request["Proxy"] == nil) then return 0 else return 403 end
* Modify lighttpd.conf to load mod_magnet and run lua code
server.modules += ( "mod_magnet" )
magnet.attract-raw-url-to = ( "/path/to/deny-proxy.lua" )
References:
https://www.kb.cert.org/vuls/id/797896
CGI web servers assign Proxy header values from client requests to
internal HTTP_PROXY environment variables
https://httpoxy.org/
httpoxy: A CGI application vulnerability
do not expose server info server_tag is configured by admin
default con->conf.server_tag is still PACKAGE_DESC, those
admin can configure server.tag otherwise.
(these changes reduce recompilation and relinking when switching
brances in source control)
Set server.stream-response-body = 1 or server.stream-response-body = 2
to have lighttpd stream response body to client as it arrives from the
backend (CGI, FastCGI, SCGI, proxy).
default: buffer entire response body before sending response to client.
(This preserves existing behavior for now, but may in the future be
changed to stream response to client, which is the behavior more
commonly expected.)
x-ref:
"fastcgi, cgi, flush, php5 problem."
https://redmine.lighttpd.net/issues/949
"Random crashing on FreeBSD 6.1"
https://redmine.lighttpd.net/issues/760
"Memory usage increases when proxy+ssl+large file"
https://redmine.lighttpd.net/issues/1283
"lighttpd+fastcgi memory problem"
https://redmine.lighttpd.net/issues/1387
move code in dynamic handlers (CGI, FastCGI, SCGI, proxy)
from *_handle_fdevent() to *_recv_response() for reuse
outside the *_handle_fdevent() routine
Set server.stream-request-body = 1 or server.stream-request-body = 2
to have lighttpd connect to backend (CGI, FastCGI, SCGI, proxy)
immediately after parsing request headers, and to stream request body
as it arrives.
default: buffer entire request body before connecting to backend,
in order to avoid tying up (limited) backend resources which are often
implemented using libraries which wait for entire request body before
proceeding.
x-ref:
"Reimplement upload (POST) handling to match apache/zeus/thttpd/boa functionality"
https://redmine.lighttpd.net/issues/376
