Commit Graph

50 Commits

Author SHA1 Message Date
Glenn Strauss 7b615d5d24 [multiple] de-dup file and piped loggers (fixes #3101)
de-dup file and piped loggers for error logs and access logs

  "RFE: de-dup file and piped loggers"
2021-09-13 03:37:11 -04:00
Glenn Strauss 309c1693ac [multiple] Y2038 32-bit signed time_t mitigations
Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.

Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t

* Y2038: lighttpd 1.4.60 and later report Y2038 safety
  $ lighttpd -V
    + Y2038 support                                    # Y2038-SAFE
  $ lighttpd -V
    - Y2038 support (unsafe 32-bit signed time_t)      # Y2038-UNSAFE

* Y2038: general platform info
  * Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
      - all major 64-bit platforms (known to this author) use 64-bit time_t
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
      - Linux x32 ABI (different from i686)
      - FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
      - NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
      - OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
      - Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
        Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
      - MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
      - e.g. OpenVMS (unknown if lighttpd builds on this platform)
  * Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
      - Linux 32-bit (including i686)
          - glibc 32-bit library support not yet available for 64-bit time_t
              - Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
                "Note: at this point, 64-bit time support in dual-time
                 configurations is work-in-progress, so for these
                 configurations, the public API only makes the 32-bit time
                 support available. In a later change, the public API will
                 allow user code to choose the time size for a given
                 compilation unit."
              - compiling with -D_TIME_BITS=64 currently has no effect
          - glibc recent (Jul 2021) mailing list discussion
      - FreeBSD i386
      - DragonFlyBSD 32-bit

* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
  * lighttpd prefers system monotonic clock instead of realtime clock
    in places where realtime clock is not required
  * lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
  * (lighttpd presumes that lighttpd will not encounter dates before 1970
    during normal operation.)
  * lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
    to convert negative timestamps for comparisions with 64-bit timestamps
    (treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
  * lighttpd provides unix_time64_t (int64_t) and
  * lighttpd provides struct unix_timespec64 (unix_timespec64_t)
    (struct timespec equivalent using unix_time64_t tv_sec member)
  * lighttpd provides gmtime64_r() and localtime64_r() wrappers
    for platforms 32-bit platforms using 32-bit time_t and
    lighttpd temporarily shifts the year in order to use
    gmtime_r() and localtime_r() (or gmtime() and localtime())
    from standard libraries, before readjusting year and passing
    struct tm to formatting functions such as strftime()
  * lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
    unsigned 32-bit and then to unix_time64_t

* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
  on 32-bit platforms using 32-bit signed time_t, underlying libraries and
  underlying filesystems might not behave properly after 32-bit signed time_t
  overflows (19 Jan 2038 03:14:08 GMT).  If a given 32-bit OS does not work
  properly using negative time_t values, then lighttpd likely will not work
  properly on that system.

* Other references and blogs
2021-09-04 08:08:26 -04:00
Glenn Strauss 0f90a9e320 [core] reduce memcmp in http_request_parse_header
extend http_header_parse_ctx to support enum http_header_h2_e
HTTP/2 pseudo-headers in hpctx->id before using memcmp()
2021-08-27 02:16:53 -04:00
Glenn Strauss d5366c0aee [core] request_config_reset()
move configfile.c:config_reset_config()
  to reqpool.c:request_config_reset()
2021-08-27 02:16:53 -04:00
Glenn Strauss 9a5e1652be [multiple] static file optimization; reuse cache
reuse cache lookup in common case of serving a static file
rather than repeating the stat_cache_entry lookup
(which is more work than memcmp() to re-check stat_cache_entry match)
2021-05-06 17:35:00 -04:00
Glenn Strauss 13ea2d880b [core] consistent inclusion of sys-time.h 2021-04-28 14:08:29 -04:00
Glenn Strauss 4a600dabd5 [mod_auth] close HTTP/2 connection after bad pass
mitigation slows down brute force password attacks

  "Possible feature: authentication brute force hardening"
2021-02-06 08:29:41 -05:00
Glenn Strauss 903024d711 [core] track Content-Length from backend (fixes #3046)
track Content-Length from backend in r->resp_body_scratchpad

  "Failure on second request in http proxy backend"
2020-12-16 02:00:17 -05:00
Glenn Strauss e9309ae6e6 [core] li_restricted_strtoint64()
(make public for reuse)
2020-12-15 22:31:08 -05:00
Glenn Strauss cd2ff4202d [core] remove unused r->content_length
(was parsed Content-Length sent by backend)
(might add back in future along with stricter validation of field value)
2020-10-13 22:31:10 -04:00
Glenn Strauss 81029b8b51 [multiple] inline chunkqueue where always alloc'd
inline struct chunkqueue where always allocated in other structs

(memory locality)
2020-10-11 12:19:27 -04:00
Glenn Strauss 1a64c9e2f7 [core] reuse r->start_hp.tv_sec for r->start_ts
(remove duplicated field from (request_st *))
2020-10-11 12:19:27 -04:00
Glenn Strauss 3fbb5773e9 [core] http_header_e <=> lshpack_static_hdr_idx
map enum http_header_e to enum lshpack_static_hdr_idx
map enum lshpack_static_hdr_idx to enum http_header_e
2020-10-11 12:19:26 -04:00
Glenn Strauss 367f30a645 [multiple] extend enum http_header_e list 2020-10-11 12:19:26 -04:00
Glenn Strauss 8fc8ab891a [core] http_request_parse_header() specialized
http_request_parse_header() specialized for HTTP/2 request headers
to be parsed as each field-name and value is HPACK-decoded; send headers
directly from HPACK decoder, rather than double-buffering in chunkqueue

http_request_headers_process_h2() for post-processing
2020-10-11 11:43:06 -04:00
Glenn Strauss ada09a23b0 [core] h2_send_headers() specialized for resp hdrs
specialized version of http_response_write_header(); send headers
directly to HPACK encoder, rather than double-buffering in chunkqueue
2020-10-11 11:43:06 -04:00
Glenn Strauss 6eca864c5e [core] h2.[ch] with stub funcs (incomplete)
(subsequently incrementally updated using git rebase)

huge props and many thank yous to writers of testing tools used while
developing HTTP/2 support in lighttpd:

  h2spec - conformance testing tool for HTTP/2 implementation
  h2load - HTTP/2 benchmarking tool
  curl   - command line tool and library for transferring data with URLs
2020-10-11 11:43:02 -04:00
Glenn Strauss 6971c6c811 [core] move http_request_headers_process()
move http_request_headers_process() to request.[ch]
2020-10-03 09:05:38 -04:00
Glenn Strauss afc2025d8e [core] reset connection counters per connection
reset connection counters per connection, not per request

adjust mod_accesslog and mod_rrdtool usage

continue to count mod_rrdtool per request rather than per connection
so that data is updated after each request, rather than aggregated
to the end of a potentially long-lived connection with many keep-alives.
2020-10-03 09:05:38 -04:00
Glenn Strauss 1041a56dd9 [core] server.feature-flags to enable h2
server.feature-flags = ("server.h2proto" => "enable") to enable h2
2020-08-13 15:05:25 -04:00
Glenn Strauss 7420526ddb [core] decode Transfer-Encoding: chunked from gw
decode Transfer-Encoding: chunked from gw (gateway backends)

Transfer-Encoding: chunked is a hop-by-hop header.

Handling chunked encoding remove a hurdle for mod_proxy to send HTTP/1.1
requests to backends and be able to handle HTTP/1.1 responses.

Other backends ought not to send Transfer-Encoding: chunked, but in
practice, some implementations do.
2020-08-02 07:47:42 -04:00
Glenn Strauss a0029b21a1 [core] remove r->uri.path_raw; generate as needed
(r->uri.path_raw previously duplicated from r->target, minus query-part)
2020-07-08 19:54:29 -04:00
Glenn Strauss d013d0abd3 [core] http_request_parse_target()
http_request_parse_target() split from http_response_prepare()
2020-07-08 19:54:29 -04:00
Glenn Strauss 9914bb297b [core] C99 restrict on some base funcs
buffer.[ch], chunk.[ch], request.[ch], sock_addr.[ch]
2020-07-08 19:54:29 -04:00
Glenn Strauss 7c7f8c467c [multiple] split con, request (very large change)
NB: r->tmp_buf == srv->tmp_buf (pointer is copied for quicker access)

NB: request read and write chunkqueues currently point to connection
    chunkqueues; per-request and per-connection chunkqueues are
    not distinct from one another
      con->read_queue  == r->read_queue
      con->write_queue == r->write_queue

NB: in the future, a separate connection config may be needed for
    connection-level module hooks.  Similarly, might need to have
    per-request chunkqueues separate from per-connection chunkqueues.
    Should probably also have a request_reset() which is distinct from
2020-07-08 19:54:29 -04:00
Glenn Strauss aca9d45adf [core] move request state into (request_st *)
NB: in the future, a separate connection state may be needed for
    connection-level state (different from request state)
2020-07-08 19:54:29 -04:00
Glenn Strauss a22cdca1cb [core] move addtl request-specific struct members 2020-07-08 19:54:29 -04:00
Glenn Strauss 100dfaa3f3 [core] move plugin_ctx into (request_st *)
NB: in the future, a separate plugin_ctx may be needed for
    connection-level plugins to keep state across multiple requests
2020-07-08 19:54:29 -04:00
Glenn Strauss 8131e4396d [core] move addtl request-specific struct members 2020-07-08 19:54:29 -04:00
Glenn Strauss 1474be7859 [core] move addtl request-specific struct members 2020-07-08 19:54:29 -04:00
Glenn Strauss 6fe031ef37 [core] move request start ts into (request_st *)
move request start timestamps into (request_st *)
2020-07-08 19:54:29 -04:00
Glenn Strauss af5df35275 [core] rename content_length to reqbody_length
rename content_length to reqbody_length in request,
to more easily differentiate request body length
from response content_length
2020-07-08 19:54:29 -04:00
Glenn Strauss 8c46768d5b [core] remove unused request_st member 'request' 2020-07-08 19:54:29 -04:00
Glenn Strauss 6870b0f55b [core] pass (request_st *) to request.c funcs
instead of (struct connection *)
2020-07-08 19:54:29 -04:00
Glenn Strauss 668cb6418f [core] move request_st,request_config to request.h 2020-07-08 19:54:29 -04:00
Glenn Strauss 6748a58cca [core] pass scheme port to http_request_parse()
con->proto_default_port is a property of the connection,
which influences the default port used in host normalization
2020-07-08 19:54:29 -04:00
Glenn Strauss 1dd58c5ad8 [multiple] con->proto_default_port 2020-07-08 19:54:28 -04:00
Glenn Strauss 034d7d6734 [core] http_request_host_policy w/ http_parseopts
pass http_parseopts arg (instead of con) to http_request_host_policy()
2020-07-08 19:54:28 -04:00
Glenn Strauss 36f3206a4c [core] pass ptr to http_request_parse() 2020-02-24 11:15:32 -05:00
Glenn Strauss 63e32e8100 [core] perf: HTTP header parsing using \n offsets 2020-02-24 11:15:32 -05:00
Glenn Strauss c22ec74770 [core] do not pass srv to http header parsing func
srv is retrieved from con->srv in the few cases where needed
2020-02-24 11:14:45 -05:00
Glenn Strauss 25185d1de0 [core] pass req hdrs buffer to http_request_parse 2019-02-08 22:49:46 -05:00
Glenn Strauss 3eb7902e10 [core] server.http-parseopts URL normalization opt (fixes #1720)
server.http-parseopts = ( ... ) URL normalization options

Note: *not applied* to CONNECT method

Note: In a future release, URL normalization likely enabled by default
  (normalize URL, reject control chars, remove . and .. path segments)
  To prepare for this change, lighttpd.conf configurations should
  explicitly select desired behavior by enabling or disabling:
    server.http-parseopts = ( "url-normalize" => "enable", ... )
    server.http-parseopts = ( "url-normalize" => "disable" )

  "lighttpd ... compares URIs to patterns in the (1) url.redirect and (2) url.rewrite configuration settings before performing URL decoding, which might allow remote attackers to bypass intended access restrictions, and obtain sensitive information or possibly modify data."
  "Rewrite/redirect rules and URL encoding"
2018-08-12 14:43:22 -04:00
Glenn Strauss 04d76e7afd [core] some header cleanup
provide standard types in first.h instead of base.h
provide lighttpd types in base_decls.h instead of settings.h
reduce headers exposed by headers for core data structures
  do not expose <pcre.h> or <stdlib.h> in headers
move stat_cache_entry to stat_cache.h
reduce use of "server.h" and "base.h" in headers
2018-04-08 22:22:23 -04:00
Glenn Strauss e6564641d8 [core] remove unused func 2018-01-19 22:13:58 -05:00
Glenn Strauss 40f72a41b9 [core] omit default port from normalized host str
omit default scheme port from normalized host string
2017-04-25 11:12:53 -04:00
Glenn Strauss 1104afc49b [core] export http_request_host_policy() for reuse 2017-04-01 19:20:09 -04:00
Glenn Strauss b47494d4cd [config] opts for http header parsing strictness (fixes #551, fixes #1086, fixes #1184, fixes #2143, #2258, #2281, fixes #946, fixes #1330, fixes #602, #1016)
server.http-parseopt-header-strict  = "enable"
server.http-parseopt-host-strict    = "enable"  (implies host-normalize)
server.http-parseopt-host-normalize = "disable"

defaults retain current behavior, which is strict header parsing
and strict host parsing, with enhancement to normalize IPv4 address
and port number strings.

For lighttpd tests, these need to be enabled (and are by default)
For marginally faster HTTP header parsing for benchmarks, disable these.

To allow
  - underscores in hostname
  - hypen ('-') at beginning of hostname
  - all-numeric TLDs
  server.http-parseopt-host-strict    = "disable"

  "lighttpd doesn't allow underscores in host names"
  "hyphen in hostname"
  "a numeric tld"
  "Numeric tld's"
  "Bad Request"
  "400 Bad Request when using Numeric TLDs"

To allow a variety of numerical formats to be converted to IP addresses
  server.http-parseopt-host-strict    = "disable"
  server.http-parseopt-host-normalize = "enable"

  "URL encoding leads to "400 - Bad Request""
  "400 Bad Request when using IP's numeric value ("ip2long()")"

To allow most 8-bit and 7-bit chars in headers
  server.http-parseopt-header-strict  = "disable"  (not recommended)

  "Russian letters not alowed?"
  "header Content-Disposition with russian '?' (CP1251, ascii code 255) causes error"
2016-05-19 19:15:13 -04:00
Glenn Strauss 8abd06a7ff consistent inclusion of config.h at top of files (fixes #2073)
From: Glenn Strauss <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-19 15:14:35 +00:00
Jan Kneschke bcdc6a3bbc moved everything below trunk/ and added branches/ and tags/
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2005-02-20 14:27:00 +00:00