Commit Graph

226 Commits

Author SHA1 Message Date
Glenn Strauss a067d99fa0 [core] chunkqueue_small_resp_optim if resp < 16k
(adjusted down from 32k after some basic benchmarking using weighttp)
2021-10-27 04:16:38 -04:00
Glenn Strauss dd22dfa74c [core] fix chunked decoding from backend (#3044, #3046)
(fix typo missed in merge of 8e6dc4fe)
2021-10-02 12:35:18 -04:00
Glenn Strauss 8e6dc4fe94 [core] fix chunked decoding from backend (#3044, #3046)
fix edge case for initial chunked data
(bug introduced in lighttpd 1.4.56)

If chunked header received without data before response headers sent,
then initial chunked data might be sent to client without chunked header
if client made an HTTP/1.1 request and the response is Transfer-Encoding
chunked and lighttpd is configured to stream the response (non-zero
value for  This might occur if lighttpd
backend is connected via a unix domain socket and the initial chunk is
large and coming from a temporary file.  It may be sent in a separate
packet since lighttpd does not use TCP_CORK on unix domain sockets.

  "Failure on second request in http proxy backend"
  "Socket errors after update to version 1.4.56"
2021-09-30 17:34:03 -04:00
Glenn Strauss 7a21b3856e [core] log_error_multiline()
rename log_error_multiline_buffer() to log_error_multiline()
and take (char *)ptr and (size_t)len instead of (buffer *)b

When debug printing request and response headers,
print each header on separate line for readability
and omit '\r' if "\r\n" ends line
2021-09-08 15:06:07 -04:00
Glenn Strauss 55d8ed1b25 [core] remove HANDLER_WAIT_FOR_FD
If a system call returns EMFILE, then admin should increase
server.max-fds and check/increase rlimits for num files (ulimit -Hn)

Alternatively, the admin might decrease server.max-connections to limit
the number of connections served in parallel.
2021-09-08 15:06:06 -04:00
Glenn Strauss 309c1693ac [multiple] Y2038 32-bit signed time_t mitigations
Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.

Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t

* Y2038: lighttpd 1.4.60 and later report Y2038 safety
  $ lighttpd -V
    + Y2038 support                                    # Y2038-SAFE
  $ lighttpd -V
    - Y2038 support (unsafe 32-bit signed time_t)      # Y2038-UNSAFE

* Y2038: general platform info
  * Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
      - all major 64-bit platforms (known to this author) use 64-bit time_t
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
      - Linux x32 ABI (different from i686)
      - FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
      - NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
      - OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
      - Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
        Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
      - MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
      - e.g. OpenVMS (unknown if lighttpd builds on this platform)
  * Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
      - Linux 32-bit (including i686)
          - glibc 32-bit library support not yet available for 64-bit time_t
              - Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
                "Note: at this point, 64-bit time support in dual-time
                 configurations is work-in-progress, so for these
                 configurations, the public API only makes the 32-bit time
                 support available. In a later change, the public API will
                 allow user code to choose the time size for a given
                 compilation unit."
              - compiling with -D_TIME_BITS=64 currently has no effect
          - glibc recent (Jul 2021) mailing list discussion
      - FreeBSD i386
      - DragonFlyBSD 32-bit

* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
  * lighttpd prefers system monotonic clock instead of realtime clock
    in places where realtime clock is not required
  * lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
  * (lighttpd presumes that lighttpd will not encounter dates before 1970
    during normal operation.)
  * lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
    to convert negative timestamps for comparisions with 64-bit timestamps
    (treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
  * lighttpd provides unix_time64_t (int64_t) and
  * lighttpd provides struct unix_timespec64 (unix_timespec64_t)
    (struct timespec equivalent using unix_time64_t tv_sec member)
  * lighttpd provides gmtime64_r() and localtime64_r() wrappers
    for platforms 32-bit platforms using 32-bit time_t and
    lighttpd temporarily shifts the year in order to use
    gmtime_r() and localtime_r() (or gmtime() and localtime())
    from standard libraries, before readjusting year and passing
    struct tm to formatting functions such as strftime()
  * lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
    unsigned 32-bit and then to unix_time64_t

* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
  on 32-bit platforms using 32-bit signed time_t, underlying libraries and
  underlying filesystems might not behave properly after 32-bit signed time_t
  overflows (19 Jan 2038 03:14:08 GMT).  If a given 32-bit OS does not work
  properly using negative time_t values, then lighttpd likely will not work
  properly on that system.

* Other references and blogs
2021-09-04 08:08:26 -04:00
Glenn Strauss 2899d7f983 [core] construct file path after docroot hook
also remove some excess/duplicative trace during request processing
2021-08-27 02:16:54 -04:00
Glenn Strauss f490078d0f [multiple] buffer_copy_string_len_lc()
convenience wrapper combining
and making a single pass over string
2021-08-27 02:16:53 -04:00
Glenn Strauss d5366c0aee [core] request_config_reset()
move configfile.c:config_reset_config()
  to reqpool.c:request_config_reset()
2021-08-27 02:16:53 -04:00
Glenn Strauss af3df29ae8 [multiple] reduce redundant NULL buffer checks
This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.

Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.

In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func.  In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.

- check for empty strings at config time and set value to NULL if blank
  string will be ignored at runtime; at runtime, simple pointer check
  for NULL can be used to check for a value that has been set and is not
  blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
  and use buffer_is_unset() instead of buffer_is_empty(),
  where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
  known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
  truncate string, and use buffer_extend() to extend

Examples where buffer known not to be NULL:
  - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
    (though we might set it to NULL if buffer_is_blank(cpv->v.b))
  - address of buffer is arg (&foo)
    (compiler optimizer detects this in most, but not all, cases)
  - buffer is checked for NULL earlier in func
  - buffer is accessed in same scope without a NULL check (e.g. b->ptr)

internal behavior change:
  callers must not pass a NULL buffer to some funcs.
  - buffer_init_buffer() requires non-null args
  - buffer_copy_buffer() requires non-null args
  - buffer_append_string_buffer() requires non-null args
  - buffer_string_space() requires non-null arg
2021-08-27 02:16:53 -04:00
Glenn Strauss a8b5ad2b5c [core] tweaks writing response header (better asm) 2021-08-27 02:16:53 -04:00
Glenn Strauss 0afab29cfb [core] merge uri_raw and uri_clean hooks
hooks are run consecutively in http_response_prepare()

merge uri_raw before uri_clean to preserve existing ordering
2021-05-12 05:23:28 -04:00
Glenn Strauss 3a9a3716c4 [core] adjust r->http_host ptr caching
faster to use http_header_request_set_ptr() and buffer_copy_string_len()
than to use http_header_request_set() and http_header_request_get()

check r->http_host for NULL instead of relying on buffer_copy_* to do so
2021-05-10 20:20:28 -04:00
Glenn Strauss 17d9733f6e [core] http_response_prepare() OPTIONS *, CONNECT
special-case OPTIONS * and CONNECT in http_response_prepare()

http_response_prepare() is no longer revisited if r->handler_module
is set, so it is no longer necessary to fill r->physical.path for
2021-05-07 21:29:02 -04:00
Glenn Strauss 325690b039 [core] mark cold paths in http_response_config
HTTP/1.1 is not typically disabled

GET/HEAD are typically the most frequent request types
  and request body is not typically present for GET/HEAD
2021-05-07 16:39:52 -04:00
Glenn Strauss 9a5e1652be [multiple] static file optimization; reuse cache
reuse cache lookup in common case of serving a static file
rather than repeating the stat_cache_entry lookup
(which is more work than memcmp() to re-check stat_cache_entry match)
2021-05-06 17:35:00 -04:00
Glenn Strauss 13ea2d880b [core] consistent inclusion of sys-time.h 2021-04-28 14:08:29 -04:00
Glenn Strauss 680e6b3bca [multiple] buffer_copy_path_len2() aggregate 2021-04-02 01:16:42 -04:00
Glenn Strauss 7a078f567a [core] mark debug path unlikely
mark debug path unlikely
remove redundant debug trace
2021-04-02 01:16:42 -04:00
Glenn Strauss dc01487ea6 [multiple] use buffer_append_* aggregates
reduces the number of round-trips into some frequently-called routines
2021-04-02 01:16:40 -04:00
Glenn Strauss a6d1dccad3 [multiple] strftime %F and %T
strftime %F for %Y-%m-%d, and %T for %H:%M:%S
2021-04-02 01:16:08 -04:00
Glenn Strauss 26f354cb37 [multiple] http_header APIs to reduce str copies 2021-03-26 22:38:36 -04:00
Glenn Strauss ea38287a4c [multiple] do not clear physical.path if finished
do not reset r->physical.path (unneeded) if returning HANDLER_FINISHED

error handler will reset r->physical.path, if necessary
2021-03-26 07:33:42 -04:00
Glenn Strauss 38c8735850 [multiple] optimize primitives, buffer_extend()
optimize buffer_* primitives

Other than buffer_string_set_length(), reallocate with one power-2 step
in size (or use the requested size, if larger).  This replaces the fixed
BUFFER_PIECE_SIZE round-up of only 64 bytes extension each reallocation,
which could lead to excessive reallocations in some scenarios.

buffer_extend() convenience routine to prep for batch append
(combines buffer_string_prepare_append() and buffer_commit())

mod_fastcgi, mod_scgi, mod_proxy and others now leverage buffer_extend()

mod_scgi directly performs little-endian encoding of short ints

http_response_write_header() optimizes writing response header,
leveraging buffer_extend()

modify mod_proxy to append line ends
similar to how it is done in http_response_write_header()
(removes one call to buffer_append_string_len())
2021-03-26 07:33:42 -04:00
Glenn Strauss f9cd50b782 [core] modify path in-place checking for path-info 2021-03-26 07:33:42 -04:00
Glenn Strauss 81ef66eaf0 [multiple] buffer_has_slash_suffix()
2021-03-26 07:33:42 -04:00
Glenn Strauss 8e093e90f5 [core] http_response_physical_path_error()
consolidate code from error paths in
http_response_physical_path_check() into
2021-03-26 07:33:42 -04:00
Glenn Strauss cc35c03c3c [core] RFC 7233 Range handling for non-streaming
RFC 7233 Range handling for all non-streaming responses,
including (non-streaming) dynamic responses

(previously Range responses handled only for static files)
2021-03-02 10:14:25 -05:00
Glenn Strauss 4a600dabd5 [mod_auth] close HTTP/2 connection after bad pass
mitigation slows down brute force password attacks

  "Possible feature: authentication brute force hardening"
2021-02-06 08:29:41 -05:00
Glenn Strauss 122094e3e3 [multiple] employ http_date.h, sys-time.h
- replace use of strptime() w/ implementation specialized for HTTP dates
- use thread-safe gmtime_r(), localtime_r() (replace localtime, gmtime)
2020-12-24 16:13:20 -05:00
Glenn Strauss d15e2286f2 [core] fix handling chunked response from backend (fixes #3044)
(thx flynn)

When the backend sends Transfer-Encoding: chunked (which is HTTP/1.1,
not HTTP/1.0) and the client is HTTP/1.1, then lighttpd will transfer
the chunked encoding as it is sent from the backend without excess data
copying involved in decoding and then re-encoding. The bug is that if
the first data chunk (or chunks) is partially received at the same time
as the response headers, then the chunked encoding got corrupted.

  "Socket errors after update to version 1.4.56"
2020-12-09 02:59:36 -05:00
Glenn Strauss 019c513819 [multiple] use http_chunk_append_file_ref()
use http_chunk_append_file_ref() and http_chunk_append_file_ref_range()

reduce resource usage (number of fds open) by reference counting open
fds to files served, and sharing the fd among FILE_CHUNKs in responses
2020-10-20 11:51:48 -04:00
Glenn Strauss fe02111888 [multiple] stat_cache_path_stat() for struct st
stat_cache_path_stat() for cached (struct st *)
2020-10-19 21:40:14 -04:00
Glenn Strauss 66d1ec485c [core,mod_deflate] leverage cache of open fd
leverage simple cache of open file in stat_cache
(use stat_cache_get_entry_open())

future: reference count fd instead of dup()
  (requires extending chunkqueue interfaces)
2020-10-19 21:40:14 -04:00
Glenn Strauss 4d6d1e790a [multiple] use light_btst() for hdr existence chk 2020-10-11 12:19:27 -04:00
Glenn Strauss ee1f7af24d [core] be more precise checking streaming flags 2020-10-11 12:19:27 -04:00
Glenn Strauss 81029b8b51 [multiple] inline chunkqueue where always alloc'd
inline struct chunkqueue where always allocated in other structs

(memory locality)
2020-10-11 12:19:27 -04:00
Glenn Strauss 86e5f09062 [core] perf adjustments to avoid load miss 2020-10-11 12:19:27 -04:00
Glenn Strauss 660d719a2a [multiple] code reuse chunkqueue_peek_data()
code reuse of chunkqueue_peek_data() and chunkqueue_read_data()
2020-10-11 12:19:27 -04:00
Glenn Strauss 362be7b3bb [core] relay 1xx from backend over HTTP/1.1
relay 1xx from backend over HTTP/1.1, e.g. 103 Early Hints
(if client is connected using HTTP/1.1)

enabled by default unless disabled in lighttpd.conf with:
  server.feature-flags += ( "server.h1-discard-backend-1xx" = "enable" )

Warning: backends which send 103 Early Hints should check User-Agent
before doing so since naive clients might not handle unexpected 1xx.
Some clients may take the 1xx response as the final response, expecting
only one response.  Some clients might not properly handle 100 Continue
if the client did not send Expect: 100-continue with the request. Security Considerations

  An HTTP Status Code for Indicating Hints (103 Early Hints)
2020-10-11 12:19:27 -04:00
Glenn Strauss 367f30a645 [multiple] extend enum http_header_e list 2020-10-11 12:19:26 -04:00
Glenn Strauss 9c8981a7d1 [core] tst,set,clr macros for r->{rqst,resp}_htags 2020-10-11 12:19:26 -04:00
Glenn Strauss 327de98b38 [core] adjust response header prep for common case 2020-10-11 12:19:26 -04:00
Glenn Strauss 5fd8a26a75 [core] defer optimization to read small files
defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured
2020-10-11 12:19:24 -04:00
Glenn Strauss ada09a23b0 [core] h2_send_headers() specialized for resp hdrs
specialized version of http_response_write_header(); send headers
directly to HPACK encoder, rather than double-buffering in chunkqueue
2020-10-11 11:43:06 -04:00
Glenn Strauss 8d7e9439b5 [core] connection transition to HTTP/2; incomplete

(basic framework with some stub functions; incomplete)
(subsequently incrementally updated using git rebase)

HTTP/2 via TLS ALPN extension  (TLS)
HTTP/2 via Upgrade: h2c        (cleartext)
HTTP/2 via Prior Knowledge     (cleartext)

server.feature-flags += ("server.h2proto" => "enable")
to enable HTTP/2
server.feature-flags += ("server.h2c" => "enable")
to enable upgrade to HTTP/2 over cleartext http
2020-10-11 11:43:06 -04:00
Glenn Strauss 2c8f1b4621 [core] isolate more resp code in response.c
http_response_handler() and response generation flow control logic
2020-10-03 09:05:39 -04:00
Glenn Strauss 77057a7c02 [core] lowercase response hdr field names for h2 2020-10-03 09:05:38 -04:00
Glenn Strauss e089facf84 [core] do not send Connection: close if h2 2020-10-03 09:05:38 -04:00
Glenn Strauss 1041a56dd9 [core] server.feature-flags to enable h2
server.feature-flags = ("server.h2proto" => "enable") to enable h2
2020-08-13 15:05:25 -04:00