Commit Graph

145 Commits

Author SHA1 Message Date
Glenn Strauss a9970fec23 [core] consolidate fork()/execve() code (#1393)
(refactoring work to address issue #1393)

  "access log pipe writer should restart child process if it exits"
2017-07-15 22:42:15 -04:00
Gaël PORTAY e8498bbfcc [core] silence compiler warnings if !HAVE_FORK
silence compiler warnings if HAVE_FORK is not set

However, if HAVE_FORK is not set, then -Werror was probably passed to
./configure, which is currently a mistake.  lighttpd can successfully
compiles src/ with -Werror on many platforms, but ./configure tests
should not be run with -Werror. [gstrauss]

github: closes #81

   "Fix warnings"
2017-05-15 22:02:33 -04:00
Glenn Strauss 7a27d5eff5 [core] buffer to disk streaming to slow backends
buffer input to disk when streaming request body to slow backends
2017-05-15 22:02:33 -04:00
Glenn Strauss 86bb8be2c8 [core] perf: skip redundant strlen() if len known
performance: skip redundant strlen() if length is already known

introduce array_get_element_klen() to take key and klen params
2017-05-15 22:02:33 -04:00
Glenn Strauss 316e959b4d [core] prep mod transitions to transparent proxy
prep mod_proxy,mod_fastcgi,mod_scgi for transition to transparent proxy
2017-05-14 00:09:23 -04:00
Glenn Strauss a48d65c8a5 [core] remove redundant resets of fde_ndx
after initialization, value of fde_ndx should be managed by fdevent.c
2017-05-14 00:09:23 -04:00
Glenn Strauss c66e826978 [mod_proxy,mod_scgi] fix truncated error trace 2017-05-10 23:21:15 -04:00
Glenn Strauss 1465cf80f9 [core] calloc plugin_config for consistent init 2017-04-30 20:38:03 -04:00
Glenn Strauss 8641d1b03f [core] no SOCK_CLOEXEC on Linux kernel < 2.6.27
Linux kernels < 2.6.27 (old!) might return EINVAL if SOCK_CLOEXEC used

2017-04-15 17:46:25 -04:00
Glenn Strauss 4796313efc [core] collect ioctl FIONREAD code
include <sys/ioctl.h> in files which use ioctl()
  instead of exposing header in local header "sys-socket.h"
2017-03-28 02:17:33 -04:00
Glenn Strauss a53f662a30 [core] remove some unused header includes
remove exposure of stdio.h in buffer.h for print_backtrace(), now static
2017-03-28 02:17:33 -04:00
Glenn Strauss aa14493e47 [tests] reduce time waiting for backends to start
reduce time spent waiting for backends to start

tests check for active listening port before proceeding

test runs now complete in about 2/3 the time
2017-03-19 23:48:50 -04:00
Glenn Strauss 1804ccbd6d [mod_scgi] do not reconnect after connect succeeds
apply diff from commit:2eaf42d0 made to mod_fastcgi.c back in 2009

(change is being made for (a bit more) consistency between the modules
 as mod_scgi.c was forked from mod_fastcgi.c a very long time ago, as
 well as to remove call to sleep for 10ms (usleep(10000) from mod_scgi)

  "fastcgi incomplete header might crash lighty"
2017-03-19 23:48:50 -04:00
Glenn Strauss 0a635fc8be [core] consolidate dynamic handler response parse
- consolidate dynamic handler HTTP response parsing code
- reduce string copies for CGI, FastCGI, SCGI, proxy response headers
- let read() signal EOF or EAGAIN instead of ioctl FIONREAD 0-data-ready
2017-03-19 23:48:50 -04:00
Glenn Strauss e4bb56222f [mod_cgi,fastcgi,scgi,proxy] fix streaming response (fixes #2796)
fix streaming response when = 2
and client catches up to stream from backend

(thx horgh)

  "mod_fastcgi can fail to read entire response from server"
2017-03-13 13:49:17 -04:00
Glenn Strauss bd77abe0f8 [config] more specific checks for array lists
More specific checks on contents of array lists.  Each module using
lists now does better checking on the types of values in the list
(strings, integers, arrays/lists)

This helps prevent misconfiguration of things like cgi.assign,
fastcgi.server, and scgi.server, where source code might be
served as static files if parenthesis are misplaced.

2017-03-08 11:42:59 -05:00
Glenn Strauss 1dd5cce3bc [mod_fastcgi,mod_scgi] consolidate backend process accounting (#2788)
consolidate backend process accounting for consistency

  "FreeBSD/1.4.45/SSL: requests getting stuck in handle-req state occasionally"
2017-02-25 12:41:21 -05:00
Glenn Strauss 23c3535efc [mod_scgi] fix potential repeated use of proc->id
host->max_id is now number of procs allocated
proc->id is used when generated unix socket name or port for proc
2017-02-25 12:41:21 -05:00
Glenn Strauss 317b7e5fc6 [mod_scgi] fix unused_procs bidirectional-links 2017-02-25 12:41:20 -05:00
Glenn Strauss fb6b4027c8 [mod_fastcgi,mod_scgi] consistent waitpid handling (fixes #2791)
more consistent waitpid() handling, consolidate similar code

If ECHILD received for a given pid, do not retry waitpid() for that pid

  "mod_fastcgi : pid {pid} 1 not found: No child processes"
2017-02-25 12:41:18 -05:00
Glenn Strauss aa923e05f6 [mod_fastcgi,mod_scgi] backend spawn EINTR retry (#2788)
When spawning backends, retry blocking connect() to backend if EINTR
received when attempting to see if backend is already running.  EINTR
might be received if a HUP or USR1 signal is received while connecting
(or SIGCHLD on systems without SA_RESTART)

(expected to occur extremely rarely, but simple to handle properly)

  "FreeBSD/1.4.45/SSL: requests getting stuck in handle-req state occasionally"
2017-02-11 12:35:49 -05:00
Glenn Strauss 3209f30d11 [core] handle if backend sends Transfer-Encoding (#2786)
It is still not a good idea for backend to send Transfer-Encoding unless
backend is mod_proxy, and mod_proxy should not currently receive chunked
response since mod_proxy sends HTTP/1.0 request.

If mod_proxy is changed to sent HTTP/1.1 request, then lighttpd would
need to check if client is HTTP/1.0 and would need to de-chunk and
remove any other transfer-codings if not supported by next-hop.

  "error 500 (mod_cgi.c.601) cgi died"
2017-02-11 11:06:31 -05:00
Glenn Strauss 7f9209c6d4 [mod_cgi] do not send "Status" back to client
"Status" from CGI/1.1 environment should not be sent back to client.

Also, do not send "Status" back to client in mod_scgi
and more precisely parse for "Status" in mod_fastcgi
2017-01-31 14:40:05 -05:00
Glenn Strauss b1405360fe apply debian/patches/spelling.patch
description: fix upstream spelling errors
author: Michael Gilbert <>
2017-01-31 14:36:15 -05:00
Glenn Strauss 8981ca0467 [core] use getaddrinfo,inet_pton vs gethostbyname (fixes #2783)
when available, use getaddrinfo(),inet_pton() instead of gethostbyname()

NOTE: behavior change: mod_scgi now listens to INADDR_LOOPBACK if "host"
      is not specified.  (Prior behavior was INADDR_ANY.)  Backends
      should not listen on potentially public IPs unless explicitly
      configured to do so.  This change matches a change to mod_fastcgi
      made in 2008.

  "gethostbyname deprecated, should use getaddrinfo"
2017-01-31 14:36:15 -05:00
Glenn Strauss 1adf1df285 remove redundant calls to end-of-request hooks
The (misnamed) connection_reset hook is always called after a request,
whether request completes or is aborted, and whether keep-alive or not,
so no needed to repeat the same function in the handle_connection_close
2017-01-14 01:06:16 -05:00
Glenn Strauss be520a8058 [mod_scgi] detect child exit, restart proactively
(instead of detecting upon a subsequent HTTP request)

(for backends spawned by mod_scgi)
2017-01-10 07:55:18 -05:00
Glenn Strauss 4d7f5737f1 [core] support Transfer-Encoding: chunked req body (fixes #2156)
support Transfer-Encoding: chunked request body in conjunction with = 0

dynamic handlers will still return 411 Length Required if = 1 or 2 (!= 0)
  since CGI-like env requires CONTENT_LENGTH be set
  (and mod_proxy currently sends HTTP/1.0 requests to backends,
   and Content-Length recommended for robust interaction with backend)

  "request: support Chunked Transfer Coding for HTTP PUT"
2016-12-16 16:58:04 -05:00
Glenn Strauss f9737e50a6 [mod_fastcgi,mod_scgi] warn if invalid "bin-path"
e.g. if /usr/bin/php-cgi does not exist

A distribution package might need to be installed:
'php-cli' Fedora package; 'php7.0-cgi' or 'php5-cgi' Debian package
2016-12-03 21:19:10 -05:00
Glenn Strauss 5bf5e1adcc fix race in dynamic handler configs (reentrancy) (fixes #2774)
(thx tobbe303)

  "CGI request not handled"
2016-11-28 12:39:37 -05:00
Stefan Bühler 5e3653dc5d [mod_scgi] fix segfault (fixes #2762) 2016-11-02 08:13:44 +01:00
Glenn Strauss b29e69e5b7 [mod_scgi] fix prefix matching to always match url
Fix mod_scgi prefix matching: match the prefix always against url,
not the absolute filepath (regardless of check-local)

(apply fix similar to commit:fe8b7e57 applied to mod_fastcgi in 2008)
2016-10-29 09:14:07 -04:00
Glenn Strauss fa67918d3e fix FastCGI, SCGI, proxy reconnect on failure
factor modules for consistent code flow for reconnect on failure
2016-10-25 20:30:17 -04:00
Glenn Strauss 1e129cce45 ignore return value from fcntl() FD_CLOEXEC
setting or removing FD_CLOEXEC flag does not fail

Also the use in mod_fastcgi and mod_scgi is in child after fork().
If the fd already happens to be 0 (should not happen in current code)
and removing the FD_CLOEXEC flag fails, then the backend will fail
to start.
2016-10-15 23:28:09 -04:00
Glenn Strauss 7fa5bfc938 consistent, shared code to create CGI env
consolidated from CGI, FastCGI, SCGI, SSI

Note: due to prior inconsistencies between the code in mod_cgi,
mod_fastcgi, mod_scgi, and mod_ssi, there are some minor behavior

CONTENT_LENGTH is now always set, even if 0
  (though CONTENT_LENGTH is never set for FASTCGI_AUTHORIZER)
PATH_INFO is created only if present, not if empty.
  (mod_fastcgi and mod_ssi previously set PATH_INFO="" (blank value))
PATH_TRANSLATED is now set if PATH_INFO is present
  (previously missing from mod_cgi and mod_ssi)

mod_ssi now sets DOCUMENT_ROOT to con->physical.basedir, like others
  (previously, mod_ssi set DOCUMENT_ROOT to con->physical.doc_root,
   which matched con->physical.basedir unless mod_alias changed basedir)
mod_ssi now sets REQUEST_URI to con->request.orig_uri, like others
  (previously, mod_ssi set REQUEST_URI to con->request.uri, which
   matched con->request.orig_uri except after redirects, error docs)
2016-10-10 13:37:36 -04:00
Glenn Strauss a661944d7e [mod_scgi] add uwsgi protocol support
Configuring the protocol is controlled with new lighttpd.conf directive:
  scgi.protocol = "scgi"   # default
  scgi.protocol = "uwsgi"

The uwsgi protocol differs from the SCGI protocol only in how the
request is encoded.  The response from the backend is handled the
same way for both SCGI and uwsgi protocols.

2016-09-25 02:05:56 -04:00
Glenn Strauss 93afda9c8e performance: use Linux extended syscalls and flags
reduce syscalls on Linux using extended syscalls and flags,
e.g. accept4(), pipe2(), O_CLOEXEC, SOCK_CLOEXEC, SOCK_NONBLOCK

github: closes #2
2016-09-24 02:23:49 -04:00
Glenn Strauss dc91e40657 dynamic handlers store debug flag in handler_ctx
(for persistence across multiple re-entries into routines upon
 receiving fdevent)

(setting module debug flag in global scope is still recommended
 since there are places where p->conf.debug is checked since
 handler_ctx might not be available at all points)
2016-09-19 20:02:02 -04:00
Glenn Strauss 40f16d52db [core] fix crash if ready events on abandoned fd (fixes #2748)
  "1.4.40/1.4.41 uploads to CGI may cause crash (SIGABRT)"
2016-08-24 15:30:11 -04:00
Glenn Strauss 27f85dbdf4 [core] proxy,scgi omit shutdown() to backend (fixes #2743)
Due to the POLLHUP behavior triggered on *BSD/Darwin, the shutdown()
had previously been limited to local connections.  If interested in
squeezing the last bits of performance out of a machine, an admin
should configure local connections to be AF_UNIX instead of AF_INET
or AF_INET6 to localhost.  The reason the shutdown() was originally
added in mod_proxy and mod_scgi was to aggressively reduce the number
of potential sockets in TIME_WAIT held by lighttpd.
(See commit:923688d2 "drain backend socket/pipe bufs upon FDEVENT_HUP",
 done for reliability given the aforementioned *BSD/Darwin behavior.)
When using AF_UNIX, the TIME_WAIT issue does not exist, ergo, the
recommendation is to use AF_UNIX for local sockets, when available.
Using AF_UNIX sockets is a better solution to eliminate TIME_WAIT
than is TCP shutdown() half-close which, as we have seen, might not
be handled well by frameworks which are more complex than basic read
request, send response, and close.

  "1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwin"
2016-08-07 13:09:21 -04:00
Glenn Strauss 666b9fd726 [core] enforce wait for POLLWR after EINPROGRESS (fixes #2744)
mod_fastcgi, mod_scgi, and mod_proxy must enforce wait for POLLWR
after EINPROGRESS or else getsockopt(fd, SOL_SOCKET, SO_ERROR, ...)
may succeed even though socket connection is not yet established,
and subsequent writev() will fail ENOTCONN.

(thx pkubaj)

 "1.4.40/41 writev failed: Socket is not connected (fastcgi,scgi,proxy)"
2016-08-07 00:42:58 -04:00
Glenn Strauss 1de652f40b [mod_proxy,mod_scgi] shutdown remote only if local (#2743)
shutdown(fd, SHUT_WR) after sending request to proxy or SCGI
only if remote is local and platform is not *BSD or Darwin.

The reason this fix is special-casing *BSD and Darwin is that the Single
Unix Specification and POSIX.1-2013 clearly specify that POLLHUP event
should be returned by poll only when the stream is no longer writable.
A half-closed socket that is still writable clearly does not match that
condition, yet that is what I am seeing on Darwin (El Capitan), and
presumably what others are seeing on *BSD, from which Apple originally
inherited the Darwin TCP stack.

Single Unix Specification (SUSv2) from 1997
(yes, that is nearly 20 years ago):

    The device has been disconnected. This event and POLLOUT are
    mutually exclusive; a stream can never be writable if a hangup has
    occurred. However, this event and POLLIN, POLLRDNORM, POLLRDBAND or
    POLLPRI are not mutually exclusive. This flag is only valid in the
    revents bitmask; it is ignored in the events member.

Updated version of The Open Group Base Specifications Issue 7
(published in 2013):

    A device has been disconnected, or a pipe or FIFO has been closed
    by the last process that had it open for writing. Once set, the
    hangup state of a FIFO shall persist until some process opens the
    FIFO for writing or until all read-only file descriptors for the
    FIFO are closed.  This event and POLLOUT are mutually-exclusive;
    a stream can never be writable if a hangup has occurred. However,
    this event and POLLIN, POLLRDNORM, POLLRDBAND, or POLLPRI are not
    mutually-exclusive. This flag is only valid in the revents bitmask;
    it shall be ignored in the events member.

  "1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwin"
2016-08-06 02:24:54 -04:00
Glenn Strauss 9af58a9716 revert 1.4.40 swap of REQUEST_URI, REDIRECT_URI (fixes #2738)
reverts part of commit:dbdab5db which swapped REQUEST_URI, REDIRECT_URI

  "mediawiki redirect loop if REQUEST_URI not orig req in 1.4.40"


REQUEST_URI and REDIRECT_URI are not part of CGI standard environment.
The reason for their existence is that PATH_INFO in CGI environment may
be different from the path in the current request.  The main reason for
this potential difference is that the URI path is normalized to a path
in the filesystem and tested against the filesystem to determine which
part is SCRIPT_NAME and which part is PATH_INFO.  In case-insensitive
filesystems, the URI might be lowercased before testing against the
filesystem, leading to loss of case-sensitive submission in any
resulting PATH_INFO.  Also, duplicated slashes "///" and directory
references "/." and "/.." are removed, including prior path component in
the case of "/..".  This might be undesirable when the information after
the SCRIPT_NAME is virtual information and there target script needs the
virtual path preserved as-is.  In that case, the target script can
re-parse REQUEST_URI (or REDIRECT_URI, as appropriate) to obtain the
unmodified information from the URI.

con->request.uri is equivalent to con->request.orig_uri unless the
request has been internally rewritten (e.g. by mod_rewrite, mod_magnet,
others), in which case con->request.orig_uri is the request made by the
client, and con->request.uri is the current URI being processed.

Historical REQUEST_URI (environment variable) lighttpd inconsistencies
- mod_cml     set REQUEST_URI to con->request.orig_uri
- mod_cgi     set REQUEST_URI to con->request.orig_uri
- mod_fastcgi set REQUEST_URI to con->request.orig_uri
- mod_scgi    set REQUEST_URI to con->request.orig_uri

- mod_ssi     set            REQUEST_URI to current con->request.uri
- mod_magnet  set MAGNET_ENV_REQUEST_URI to current con->request.uri
              and MAGNET_ENV_REQUEST_ORIG_URI to con->request.orig_uri

Historical REDIRECT_URI (environment variable) previously set only in
mod_fastcgi and mod_scgi, and set to con->request.uri

Since lighttpd 1.4.40 provides REDIRECT_URI with con->request.orig_uri,
changes were made to REQUEST_URI for consistency, with the hope that
there would be little impact to existing configurations since the
request uri and original request uri are the same unless there has been
an internal redirect.  It turns out that various PHP frameworks use
REQUEST_URI and require that it be the original URI requested by client.

Therefore, this change is being reverted, and lighttpd will set
REQUEST_URI to con->request.orig_uri in mod_cgi, mod_fastcgi, mod_scgi
as was done in lighttpd 1.4.39 and earlier.  Similarly, REDIRECT_URI
also has the prior behavior in mod_fastcgi and mod_scgi, and added to

A future release of lighttpd might change mod_ssi to be consistent with
the other modules in setting REQUEST_URI to con->request.orig_uri and to
add REDIRECT_URI, when an internal redirect has occurred.
2016-07-23 02:13:41 -04:00
Glenn Strauss 779c133c16 [security] do not emit HTTP_PROXY to CGI env
Strip bogus "Proxy" header before creating subprocess environment.
(mod_cgi, mod_fastcgi, mod_scgi, mod_ssi, mod_proxy)

Do not emit HTTP_PROXY to subprocess environment.
Some executables use HTTP_PROXY to configure outgoing proxy.

This is not a lighttpd security issue per se, but this change to
lighttpd adds a layer of defense to protect backend processes which
might be vulnerable due to blindly using this untrusted environment
variable.  The HTTP_PROXY environment variable should not be trusted
by a program running in a CGI-like environment.

Mitigation in lighttpd <= 1.4.40 is to reject requests w/ Proxy header:

* Create "/path/to/deny-proxy.lua", read-only to lighttpd, with content:
  if (lighty.request["Proxy"] == nil) then return 0 else return 403 end
* Modify lighttpd.conf to load mod_magnet and run lua code
    server.modules += ( "mod_magnet" )
    magnet.attract-raw-url-to = ( "/path/to/deny-proxy.lua" )

CGI web servers assign Proxy header values from client requests to
internal HTTP_PROXY environment variables
httpoxy: A CGI application vulnerability
2016-07-19 01:22:33 -04:00
Glenn Strauss c1af146a6e [mod_fastcgi,mod_scgi] check for spawning on same unix socket (#319)
error out if duplicated unix socket path and different bin-path is
used by lighttpd to start the backend.  This is an error.
Updates commits 97556d99 and b67ff2aa

  "Should warn if two FastCGI servers point to same socket file"
2016-07-10 17:24:39 -04:00
Glenn Strauss b67ff2aaca [mod_fastcgi,mod_scgi] check for spawning on same unix socket (#319)
silently use same 'host' if unix socket path is duplicated
(instead of erroring out).  Updates commit 97556d99

(It is possible for use of += in config to result in duplicated
 fastcgi.server and scgi.server entries)

  "Should warn if two FastCGI servers point to same socket file"
2016-07-10 15:08:28 -04:00
Glenn Strauss 69ec5728f9 use con->conf.server_tag in modules
do not expose server info server_tag is configured by admin

default con->conf.server_tag is still PACKAGE_DESC, those
admin can configure server.tag otherwise.

(these changes reduce recompilation and relinking when switching
 brances in source control)
2016-06-25 18:52:06 -04:00
Glenn Strauss 8dcbd61a45 [cygwin] fix mod_proxy and mod_fastcgi ioctl use
cygwin does not support ioctl on sockets, returning EOPTNOTSUPP
(would be better if cygwin used Windows ioctlsocket() instead)

Windows uses signed (socklen_t), so add some casts to quiet warnings

Windows path handling is convoluted, so disable one tests in mod_fastcgi
since trailing spaces are removed from URL for _WIN32 and __CYGWIN__ in
2016-06-23 15:46:44 -04:00
Glenn Strauss 1a18669d53 chunkqueue_append_chunkqueue()
(simpler than chunkqueue_steal() for transferring entire chunkqueue)
2016-06-21 15:33:18 -04:00
Glenn Strauss 5863cb5752 defer choosing "Transfer-Encoding: chunked"
defer choosing "Transfer-Encoding: chunked" until response header
is about to be written
2016-06-19 23:34:16 -04:00