Commit Graph

124 Commits (3dd3cde902bb45165985f587cae590df2e9dfc38)

Author SHA1 Message Date
Glenn Strauss 3dd3cde902 [core] abstraction layer for HTTP header manip
convert existing calls to manip request/response headers
convert existing calls to manip environment array (often header-related)
2018-09-23 18:01:58 -04:00
Glenn Strauss d161f53de0 [core] security: use-after-free invalid Range req
(thx Marcus Wengelin)
2018-08-12 14:43:22 -04:00
Glenn Strauss 3eb7902e10 [core] server.http-parseopts URL normalization opt (fixes #1720)
server.http-parseopts = ( ... ) URL normalization options

Note: *not applied* to CONNECT method

Note: In a future release, URL normalization likely enabled by default
  (normalize URL, reject control chars, remove . and .. path segments)
  To prepare for this change, lighttpd.conf configurations should
  explicitly select desired behavior by enabling or disabling:
    server.http-parseopts = ( "url-normalize" => "enable", ... )
    server.http-parseopts = ( "url-normalize" => "disable" )

  "lighttpd ... compares URIs to patterns in the (1) url.redirect and (2) url.rewrite configuration settings before performing URL decoding, which might allow remote attackers to bypass intended access restrictions, and obtain sensitive information or possibly modify data."
  "Rewrite/redirect rules and URL encoding"
2018-08-12 14:43:22 -04:00
Glenn Strauss c56b21084e [core] http_kv.[ch] method, status, version str
move method, status, version strings from keyvalue.[ch] to http_kv.[ch]
2018-08-05 03:44:15 -04:00
Glenn Strauss 04d76e7afd [core] some header cleanup
provide standard types in first.h instead of base.h
provide lighttpd types in base_decls.h instead of settings.h
reduce headers exposed by headers for core data structures
  do not expose <pcre.h> or <stdlib.h> in headers
move stat_cache_entry to stat_cache.h
reduce use of "server.h" and "base.h" in headers
2018-04-08 22:22:23 -04:00
Glenn Strauss cb371557e5 [core] merge redirect/rewrite pattern substitution
merge redirect/rewrite pattern substitution function (share code)
2018-01-10 01:39:05 -05:00
Glenn Strauss e8226c11cb [core] do not reparse request if async cb
do not reparse request if async callback, e.g. for mod_auth
2018-01-01 17:06:05 -05:00
Glenn Strauss 7b2514cdad [core] quiet pedantic cc warning for excess comma 2017-11-02 00:41:53 -04:00
Glenn Strauss 142971a80c [core] consolidate backend network write handlers
network_write.[ch] isolates various write, writev, sendfile wrappers
2017-11-02 00:41:53 -04:00
Glenn Strauss 9287c87dcd [core] cleanup: consolidate FAM code in stat_cache 2017-10-29 22:37:29 -04:00
Glenn Strauss 6e87da7195 [core] cleanup unused ifndef 2017-10-29 12:12:55 -04:00
Glenn Strauss 1367f60626 [core] isolate sock_addr manipulation 2017-10-29 01:23:19 -04:00
Glenn Strauss 9030cfaecf [core] SIGCHLD handle_waitpid hook for modules
centralize most waitpid() handling in core server, with hooks for
modules to be informed of pid and status when a process exits.

This enables faster discovery (and restart) of exited processes,
and also allows for lighttpd to manage backend processes in the
parent (master) process when server.max-worker > 0.
2017-09-10 16:24:34 -04:00
Glenn Strauss 3c8afd194c [core] base_decls.h to quiet compiler warnings
quiet compiler warning for -Wtypedef-redefinition
(redefinition of typedef is a C11 feature)
2017-07-30 23:45:01 -04:00
Glenn Strauss 5248b46c95 [core] sock_addr_from_str_hints reusable name res
[core] sock_addr_from_str_hints() reusable name resolution func
2017-07-15 22:42:15 -04:00
Glenn Strauss d15ddcb6fa [core] server.socket-perms to set perms on unix (fixes #656)
server.socket-perms = "0770" to set perms on unix domain socket
on which lighttpd listens for requests, e.g. $SERVER["socket"] == "..."

  "Feature request: add server config for setting permissions on Unix domain socket"
2017-06-13 08:56:13 -04:00
Glenn Strauss e685ef70f1 [core] act as transparent proxy after con Upgrade 2017-05-14 00:09:23 -04:00
Glenn Strauss 036d3d3d66 [mod_proxy] simple host/url mapping in headers (fixes #152)
Provide a simple mechanism for mapping host and urlpath header strings
in proxied request and response well-known headers.  This *is not*
intended as a one-size-fits-all, infinitely extensible, regex rewriting
engine.  Instead, the proxy.header directive aims to provide built-in
functionality in mod_proxy for a few common use cases by performing
simple host matching or urlpath prefix matching, and using the
mapping of the first match.  More complex use cases could possibly be
handled by a custom lighttpd module (which does not currently exist).

Note: the contents of the HTTP request-line and HTTP headers may or
may not be in normalized canonical forms, which may or may not influence
the simple matching performed.  Admins should take care to provide safe
defaults (fail closed) if mapping is expected to occur and blindly
passing non-mapped requests is undesirable.

proxy.header = (
    #"map-host-request" => (
        #"-" => "...",#replace provided given Host request authority
        #"..." => "-",#preserve existing authority (no further matching)
        #"..." => "", #preserve existing authority (no further matching)
        #             #(equivalent to "xxx" => "xxx")
        #"xxx" => "yyy", #map one string ("xxx") to another ("yyy")
    #"map-host-response" => (
        #"-" => "...",#replace authority used in backend request
        #"..." => "-",#replace with original authority
        #"..." => "", #preserve existing authority (no further matching)
        #             #(equivalent to "xxx" => "xxx")
        #"xxx" => "yyy", #map one string ("xxx") to another ("yyy")
    #"map-urlpath" => (
        #"/xxx"  => "/yyy",#map one urlpath prefix to another
        #"/xxx/" => "/",   #map one urlpath prefix to another
        #"/xxx"  => "",    #map one urlpath prefix to another
        #"/key"  => "/value",
        # Note: request headers have matching "key" prefix replaced with
        # "value", and response headers have matching "value" prefix
        # replaced with "key", with a pre-test of the "value" from the
        # first-matched "key" in request headers (if there was a match)
    #"https-remap" => "enable",
        # For https requests from client, map https:// to http://
        # when map-host-request matches URI in request, and map http://
        # to https:// when map-host-response matches URI in response.
        # (mod_proxy currently sends all backend requests as http)

  "feature to remove part of the URI when passing along requests..."
2017-05-07 14:34:32 -04:00
Glenn Strauss ba679290f9 [core] inet_ntop_cache now 4-element cache
(round-robin replacement; fixes what was effectively a 1-element cache)
2017-04-15 03:42:28 -04:00
Glenn Strauss e33ec75999 [core] save connection-level proto in con->proto
The per-request scheme starts with con->proto (e.g. "http")
and can later be changed per-request by mod_extforward or mod_magnet
2017-04-06 00:22:42 -04:00
Glenn Strauss e116479731 [core] make stat_cache private to stat_cache.c 2017-03-28 02:17:33 -04:00
Glenn Strauss ba953cdf45 [core] include "fdevent.h" where needed
(instead of providing #include "fdevent.h" in base.h)
2017-03-28 02:17:33 -04:00
Glenn Strauss a8561450a5 [core] server.syslog-facility (fixes #2800)
server.syslog-facility = "daemon"

  "lighttpd needs config option for syslog facility"
2017-03-19 23:48:50 -04:00
Glenn Strauss 76ad82434f [mod_proxy] remove use of con->got_response 2017-03-15 13:19:18 -04:00
Glenn Strauss 0ec41d6d80 [core] remove unused member con->in_joblist 2017-03-15 13:16:13 -04:00
Glenn Strauss 8f651a2b30 [core] config opt to intercept dynamic handler err (fixes #974)
new directive server.error-intercept = [ "enable" | "disable" ]
to intercept 4xx and 5xx responses from dynamic handlers
(e.g. CGI, FastCGI, SCGI, proxy)

Intercepted HTTP error status are then handled by one of
(if configured)

Do not use server.error-intercept with locations handled by mod_webdav!

  "would like something similar to nginx proxy_intercept_errors"
2017-02-11 14:59:15 -05:00
Glenn Strauss 3209f30d11 [core] handle if backend sends Transfer-Encoding (#2786)
It is still not a good idea for backend to send Transfer-Encoding unless
backend is mod_proxy, and mod_proxy should not currently receive chunked
response since mod_proxy sends HTTP/1.0 request.

If mod_proxy is changed to sent HTTP/1.1 request, then lighttpd would
need to check if client is HTTP/1.0 and would need to de-chunk and
remove any other transfer-codings if not supported by next-hop.

  "error 500 (mod_cgi.c.601) cgi died"
2017-02-11 11:06:31 -05:00
Glenn Strauss acc37c1cbc [mod_openssl] use TLS SNI to set host-based certs
and then allow HTTP Host header to set con->uri.authority

remove con->tlsext_server_name
2017-01-31 14:36:15 -05:00
Glenn Strauss 8960633dc7 [mod_openssl] move openssl config into mod_openssl
move openssl data structures and config parsing into mod_openssl
2017-01-31 14:36:10 -05:00
Glenn Strauss cb7ed13621 [core] new plugin hooks to help isolate SSL
handle_request_env        (called on demand by handlers to populate env)
handle_connection_shut_wr (was handle_connection_close)
handle_connection_close   (now occurs at socket close())
2017-01-14 01:06:16 -05:00
Glenn Strauss 2bc94dee82 [core] con interface for read/write; isolate SSL 2017-01-14 01:06:16 -05:00
Glenn Strauss b03c496298 [TLS] = "disable" for low mem (fixes #2778)
new directive = "enable"/"disable" to control
SSL_CTX_set_read_ahead().  Default "enable".  The "disable" setting
is intended for use on low memory systems with a slow CPU which is
unable to keep up with decryption of large request bodies.

  "larger memory usage for file uploads via SSL on embedded system"
2017-01-10 15:59:50 -05:00
Glenn Strauss 4d7f5737f1 [core] support Transfer-Encoding: chunked req body (fixes #2156)
support Transfer-Encoding: chunked request body in conjunction with = 0

dynamic handlers will still return 411 Length Required if = 1 or 2 (!= 0)
  since CGI-like env requires CONTENT_LENGTH be set
  (and mod_proxy currently sends HTTP/1.0 requests to backends,
   and Content-Length recommended for robust interaction with backend)

  "request: support Chunked Transfer Coding for HTTP PUT"
2016-12-16 16:58:04 -05:00
Glenn Strauss 544ccee5e1 [core] remove srv->entropy[]
unlikely to provide any real additional benefit as long as
PRNG has been appropriately initialized with random data
2016-12-09 01:40:46 -05:00
Glenn Strauss 1f3ad401ba [mod_deflate] skip deflate if loadavg too high (fixes #1505)
[mod_deflate] skip deflate if 1 min loadavg too high
deflate.max-loadavg  = "3.50"  # express value as string of float num

[mod_compress] skip compression if 1 min loadavg too high
compress.max-loadavg = "3.50"  # express value as string of float num

Feature available on BSD-like systems which have getloadavg() in libc

Note: load average calculations are different on different operating
systems and different types of system loads, so there is no value that
can be recommended for one-size-fits-all.

  "Enable mod_compress to abandon compression when load average is too high"
2016-10-19 16:38:47 -04:00
Glenn Strauss 7f4e156e5f [core] rand.[ch] to use better RNGs when available
prefer RAND_pseudo_bytes() (openssl), arc4random() or jrand48(),
if available, over rand()

These are not necessarily cryptographically secure, but should be better
than rand()
2016-10-15 23:28:09 -04:00
Glenn Strauss 1018ff9922 [core] server.max-request-field-size (fixes #2130)
limits total size per request of request headers submitted by client

default limit set to 8k (prior lighttpd <= 1.4.41 hard-coded 64k limit)

(similar to Apache directive LimitRequestFieldSize)

  "limits the size of HTTP request header"
2016-10-06 00:18:07 -04:00
Glenn Strauss 6ec66c4dce [core] better DragonFlyBSD support (fixes #2746)
(thx xenu)

  "[PATCH] better DragonFlyBSD support; fix crash"
2016-08-20 14:19:10 -04:00
Glenn Strauss 5e76b284df [mod_accesslog] %a %A %C %D %k %{}t %{}T (fixes #1145, fixes #1415, fixes #2081)
add support for additional commonly-used accesslog format flags

  "mod_accesslog cookie field support %{VARNAME}C"
  "access_log : %D time used in ms (not supported)"
  "%{format}t support"
2016-07-12 23:03:16 -04:00
Glenn Strauss fe02be7e34 [core] make server.max-request-size scopeable (fixes #1901)
  "make server.max-request-size scopeable"
2016-07-09 11:01:13 -04:00
Glenn Strauss 695c8f4e07 [config] config options to stream request/response (#949, #376)
This allows admin to configure if response is collected in entirety
prior to sending data to client

For compatibility with existing configs, default is existing behavior:
  buffer entire response prior to sending data to client

The following are config options, though not all implemented yet

// default: buffer entire request body before connecting to backend = 0

// stream request body to backend; buffer to temp files = 1

// stream request body to backend; minimal buffering might block upload = 2

// default: buffer entire response body before sending to client = 0

// stream response body to client; buffer to temp files = 1

// stream response body to client; minimal buffering might block backend = 2

  "fastcgi, cgi, flush, php5 problem."
 "Reimplement upload (POST) handling to match apache/zeus/thttpd/boa functionality"
2016-06-19 23:34:15 -04:00
Glenn Strauss 4eeeb8fc76 [config] server.bsd-accept-filter option
BSD accept() filters

server.bsd-accept-filter = ""           (default)
server.bsd-accept-filter = "httpready"
server.bsd-accept-filter = "dataready"

Note: this is a behavior change from prior versions.
The default is now no additional accept() filter, whereas prior
versions unconditionally enabled "httpready" accept() filter

Additionally, server.defer-accept (Linux) is inherited from global scope
into $SERVER["socket"] blocks

github: closes #65
2016-06-04 18:59:03 -04:00
Glenn Strauss b47494d4cd [config] opts for http header parsing strictness (fixes #551, fixes #1086, fixes #1184, fixes #2143, #2258, #2281, fixes #946, fixes #1330, fixes #602, #1016)
server.http-parseopt-header-strict  = "enable"
server.http-parseopt-host-strict    = "enable"  (implies host-normalize)
server.http-parseopt-host-normalize = "disable"

defaults retain current behavior, which is strict header parsing
and strict host parsing, with enhancement to normalize IPv4 address
and port number strings.

For lighttpd tests, these need to be enabled (and are by default)
For marginally faster HTTP header parsing for benchmarks, disable these.

To allow
  - underscores in hostname
  - hypen ('-') at beginning of hostname
  - all-numeric TLDs
  server.http-parseopt-host-strict    = "disable"

  "lighttpd doesn't allow underscores in host names"
  "hyphen in hostname"
  "a numeric tld"
  "Numeric tld's"
  "Bad Request"
  "400 Bad Request when using Numeric TLDs"

To allow a variety of numerical formats to be converted to IP addresses
  server.http-parseopt-host-strict    = "disable"
  server.http-parseopt-host-normalize = "enable"

  "URL encoding leads to "400 - Bad Request""
  "400 Bad Request when using IP's numeric value ("ip2long()")"

To allow most 8-bit and 7-bit chars in headers
  server.http-parseopt-header-strict  = "disable"  (not recommended)

  "Russian letters not alowed?"
  "header Content-Disposition with russian '?' (CP1251, ascii code 255) causes error"
2016-05-19 19:15:13 -04:00
Glenn Strauss 98acff0ea0 [core] add default modules while processing server config
(instead of doing separately, before processing server config)
2016-05-10 22:32:03 -04:00
Glenn Strauss 1ca52fdce3 build with libressl
libressl defines SSL_OP_NO_SSLv2 and SSL_OP_NO_SSLv3 as 0x0
  (thx Christian Heckendorf)

libressl matches ERR_remove_thread_state() signature from openssl 1.0.2
  (libressl pretends that libressl is openssl version 2.0.0,
   but openssl 1.1.0 changes signature of ERR_remove_thread_state())

libressl does not yet provide compatibility interfaces for the new
  prototypes introduced in openssl 1.1.0, including
  DH_set0_pqg() and DH_set_length()

remove OPENSSL_NO_KRB5 from build config (added in 5fab991b in 2005)
  (define USE_OPENSSL_KERBEROS if required)
  (Note: OPENSSL_NO_KRB5 removed in openssl 1.1.0)
2016-05-07 12:50:41 -04:00
Glenn Strauss dbdab5dbc9 [core] server.error-handler new directive for error pages (fixes #2702)
server.error-handler preserves HTTP status error code when error page
is static, and allows dynamic handlers to change HTTP status code
when error page is provided by dynamic handler.  server.error-handler
intercepts all HTTP status codes >= 400 except when the content is
generated by a dynamic handler (cgi, ssi, fastcgi, scgi, proxy, lua).
The request method is unconditionally changed to GET for the request
to service the error handler, and the original request method is
later restored (for logging purposes).  request body from the
original request, if present, is discarded.

server.error-handler is somewhat similar to server.error-handler-404,
but server.error-handler-404 is now deprecated, intercepts only 404
and 403 HTTP status codes, and returns 200 OK for static error pages,
a source of confusion for some admins.  On the other hand, the new
server.error-handler, when set, will intercept all HTTP status error
codes >= 400.  server.error-handler takes precedence over
server.error-handler-404 when both are set.

NOTE: a major difference between server.error-handler and the
now-deprecated server.error-handler-404 is that the values of the
non-standard CGI environment variables REQUEST_URI and REDIRECT_URI
have been swapped.  Since REDIRECT_STATUS is the original HTTP
status code, REDIRECT_URI is now the original request, and REQUEST_URI
is the current request (e.g. the URI/URL to the error handler).
The prior behavior -- which reversed REQUEST_URI and REDIRECT_URI values
from those described above -- is preserved for server.error-handler-404.

Additionally, REDIRECT_STATUS is now available to mod_magnet, which
continues to have access to request.uri and request.orig_uri.

See further discussion at

github: closes #36
2016-04-25 01:01:08 -04:00
Glenn Strauss 87b172e70e remove unused con->error_handler member
Also remove con->in_error_handler member since non-zero
con->error_handler_saved_status can be used as flag to
indicate the same thing
2016-04-25 01:01:08 -04:00
Glenn Strauss 71ed1912c7 [config] server.listen-backlog option (fixes #1825, #2116)
See doc/config/lighttpd.conf for explanation of listen() backlog queue

Additionally, mod_fastcgi and mod_scgi backend servers can now also be
configured with separate listen-backlog settings per server

  "add server.listen-backlog option instead of hard-coded value (128 * 8) for listen()"
  "Don't disable backend when overloaded"

Closes #50
2016-04-18 04:29:28 -04:00
Glenn Strauss d17d48e01e [stat] mimetype.xattr-name global config option (fixes #2631)
For backwards compatibility with existing lighttpd configs, default is
  mimetype.xattr-name = "Content-Type"

Those who wish to use the definition of xattr mimetype
can set the following in the global lighttpd config:
  mimetype.xattr-name = "user.mime_type"

From: Glenn Strauss <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-26 13:49:43 +00:00
Glenn Strauss 292309f88b [core] lighttpd -tt performs preflight startup checks (fixes #411)
lighttpd -t loads config file and performs syntax check
lighttpd -tt (new) performs preflight startup checks,
  including loading and initializing modules, but skipping any
  potentially destructive actions which might affect an already
  running server (separate instance).  These currently include:
  - skipping pidfile modification
  - skipping bind() to network sockets
  - skipping open of error and access logs

From: Glenn Strauss <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-26 13:39:54 +00:00