Commit Graph

126 Commits

Author SHA1 Message Date
Glenn Strauss c22a56fe3b [core] restart dead proc on connect error if local
restart dead proc on connect error only if proc started by lighttpd
2021-10-27 04:16:38 -04:00
Glenn Strauss 14b8d90288 [core] separate func for gw_authorizer_ok() 2021-10-27 04:16:38 -04:00
Glenn Strauss 41c3cc0965 [core] disable server.graceful-restart-bg if spawn
disable server.graceful-restart-bg if spawning gateway backends
via *.server "bin-path"
2021-10-01 06:39:47 -04:00
Glenn Strauss 6e45cff05a [core] disable streaming response with authorizer (fixes #3106)
disable streaming response while processing "authorizer" mode
until "authorizer" response 200 OK from the backend is complete

(thx jefftharris)

  "FastCGI authorizer hang with"
  "FastCGI authorizer hang with"
2021-10-01 06:39:23 -04:00
Glenn Strauss bee35b4877 [core] improve chunk buffer reuse from backends
mark and temporarily defer event handling of backend events in order
to handle at same time as the associated front-end connection events
2021-09-30 17:34:03 -04:00
Glenn Strauss f19f71625c [multiple] internal control for backend read bytes
separate internal control for backend max_per_read

When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a smaller buffer to help reduce memory usage.

When not streaming, attempt to read and empty kernel socket bufs.
  (e.g. MAX_READ_LIMIT 256k)

When writing to sockets (or pipes) attempt to fill kernel socket bufs.
  (e.g. MAX_WRITE_LIMIT 256k)
2021-09-28 11:05:55 -04:00
Glenn Strauss b38817b678 [core] mark attr malloc, returns nonnull
mark some core code with attributes malloc, returns nonnull
2021-09-17 05:06:16 -04:00
Glenn Strauss 6ffabc9607 [core] default backend "connect-timeout" to 8 (#3086)
default backend "connect-timeout" to 8 seconds

Though this is is a behavior change where there previously was no
timeout, this is configurable by lighttpd.conf admin, and having a
default connection timeout of a fairly large value (8 seconds) puts
a (default) limit on resource usage waiting for socket connect().

  "sockets disabled, out-of-fds with proxy module"
2021-09-16 04:40:13 -04:00
Glenn Strauss a3e9faa479 [multiple] quiet coverity warnings 2021-09-09 02:16:21 -04:00
Glenn Strauss f364c8ef36 [core] remove redundant waitpid() on each backend
remove redundant waitpid() on each backend
(performed each second during periodic maintenance)

Doing so is redundant with handle_waitpid hook since lighttpd 1.4.46
2021-09-08 15:06:07 -04:00
Glenn Strauss e77bc3995c [core] reduce wait time in gw_spawn_connection()
reduce wait time in gw_spawn_connection() from 10ms down to 1ms.
That is still plenty of time to detect immediate execve()/clone()
2021-09-08 15:06:07 -04:00
Glenn Strauss 90978eb808 [core] mark cold more gateway maintenance code 2021-09-08 15:06:07 -04:00
Glenn Strauss 9c7ef4f064 [core] consolidate more gw_host, gw_proc init code
clearer code: host->num_procs as num gw_proc in host->first linked list
2021-09-08 15:06:07 -04:00
Glenn Strauss 86c87d3c6d [core] use mod name in trace instead of mod_gw 2021-09-08 15:06:07 -04:00
Glenn Strauss 55d8ed1b25 [core] remove HANDLER_WAIT_FOR_FD
If a system call returns EMFILE, then admin should increase
server.max-fds and check/increase rlimits for num files (ulimit -Hn)

Alternatively, the admin might decrease server.max-connections to limit
the number of connections served in parallel.
2021-09-08 15:06:06 -04:00
Glenn Strauss 323e03fb2d [core] remove connection_list_append()
merge connection_list_append() into connection_fdwaitqueue_append()

(not converted to singly-linked-list since fdwaitqueue is not expected
to be used during normal operation (except extreme overload condition),
so use allocated list of pointers (allocated when needed) instead of
adding ptr member to (every) allocated struct connection)

remove connections-glue.c
remove inclusion of connections.h by non-base files
2021-09-08 15:06:06 -04:00
Glenn Strauss 6515f77665 [core] clarify error message in gw_backend.c
clarify error message in gw_backend.c if connect() to unix socket fails
2021-09-08 15:06:06 -04:00
Glenn Strauss 02646ea2ad [core] connect, write, read timeouts on backends (fixes #3086)
implement connect(), write(), read() timeouts on backends


  "sockets disabled, out-of-fds with proxy module"
2021-09-08 15:06:06 -04:00
Glenn Strauss 4f96dac841 [core] gw_backend_error() shared code 2021-09-04 08:08:27 -04:00
Glenn Strauss ae149c42e5 [core] store time for last r/w to a backend socket 2021-09-04 08:08:26 -04:00
Glenn Strauss 9a2404cec6 [core] quiet compiler warnings
cast away signedness warning in request_check_hostname()
mod_gnutls https_add_ssl_client_entries crts allocated if crt_size != 0
  (which is already checked earlier in routine)

report from FaceBook Infer static analysis tool (
- quiet dead store warnings
- check return != NULL from allocation funcs
2021-09-04 08:08:26 -04:00
Glenn Strauss 309c1693ac [multiple] Y2038 32-bit signed time_t mitigations
Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.

Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t

* Y2038: lighttpd 1.4.60 and later report Y2038 safety
  $ lighttpd -V
    + Y2038 support                                    # Y2038-SAFE
  $ lighttpd -V
    - Y2038 support (unsafe 32-bit signed time_t)      # Y2038-UNSAFE

* Y2038: general platform info
  * Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
      - all major 64-bit platforms (known to this author) use 64-bit time_t
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
      - Linux x32 ABI (different from i686)
      - FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
      - NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
      - OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
      - Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
        Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
      - MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
      - e.g. OpenVMS (unknown if lighttpd builds on this platform)
  * Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
      - Linux 32-bit (including i686)
          - glibc 32-bit library support not yet available for 64-bit time_t
              - Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
                "Note: at this point, 64-bit time support in dual-time
                 configurations is work-in-progress, so for these
                 configurations, the public API only makes the 32-bit time
                 support available. In a later change, the public API will
                 allow user code to choose the time size for a given
                 compilation unit."
              - compiling with -D_TIME_BITS=64 currently has no effect
          - glibc recent (Jul 2021) mailing list discussion
      - FreeBSD i386
      - DragonFlyBSD 32-bit

* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
  * lighttpd prefers system monotonic clock instead of realtime clock
    in places where realtime clock is not required
  * lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
  * (lighttpd presumes that lighttpd will not encounter dates before 1970
    during normal operation.)
  * lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
    to convert negative timestamps for comparisions with 64-bit timestamps
    (treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
  * lighttpd provides unix_time64_t (int64_t) and
  * lighttpd provides struct unix_timespec64 (unix_timespec64_t)
    (struct timespec equivalent using unix_time64_t tv_sec member)
  * lighttpd provides gmtime64_r() and localtime64_r() wrappers
    for platforms 32-bit platforms using 32-bit time_t and
    lighttpd temporarily shifts the year in order to use
    gmtime_r() and localtime_r() (or gmtime() and localtime())
    from standard libraries, before readjusting year and passing
    struct tm to formatting functions such as strftime()
  * lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
    unsigned 32-bit and then to unix_time64_t

* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
  on 32-bit platforms using 32-bit signed time_t, underlying libraries and
  underlying filesystems might not behave properly after 32-bit signed time_t
  overflows (19 Jan 2038 03:14:08 GMT).  If a given 32-bit OS does not work
  properly using negative time_t values, then lighttpd likely will not work
  properly on that system.

* Other references and blogs
2021-09-04 08:08:26 -04:00
Glenn Strauss f1e8a82f1a [multiple] inline struct in con->dst_addr_buf
(mod_extforward recently changed to use buffer_move() to save addr
 instead of swapping pointers)
2021-08-27 02:16:54 -04:00
Glenn Strauss d38d5b4b48 [core] gw_host_get() comment out devel debugging
gw_host_get() comment out devel debugging trace
use variables within smaller scopes
2021-08-27 02:16:53 -04:00
Glenn Strauss b33d779865 [core] calculate backend host gw_hash at startup
- calculate backend host gw_hash at startup
- modify hash calculation at runtime to combine xor bits
  for hash value used in balancing selection
2021-08-27 02:16:53 -04:00
Glenn Strauss af3df29ae8 [multiple] reduce redundant NULL buffer checks
This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.

Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.

In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func.  In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.

- check for empty strings at config time and set value to NULL if blank
  string will be ignored at runtime; at runtime, simple pointer check
  for NULL can be used to check for a value that has been set and is not
  blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
  and use buffer_is_unset() instead of buffer_is_empty(),
  where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
  known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
  truncate string, and use buffer_extend() to extend

Examples where buffer known not to be NULL:
  - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
    (though we might set it to NULL if buffer_is_blank(cpv->v.b))
  - address of buffer is arg (&foo)
    (compiler optimizer detects this in most, but not all, cases)
  - buffer is checked for NULL earlier in func
  - buffer is accessed in same scope without a NULL check (e.g. b->ptr)

internal behavior change:
  callers must not pass a NULL buffer to some funcs.
  - buffer_init_buffer() requires non-null args
  - buffer_copy_buffer() requires non-null args
  - buffer_append_string_buffer() requires non-null args
  - buffer_string_space() requires non-null arg
2021-08-27 02:16:53 -04:00
Glenn Strauss 1553dc7be8 [core] reduce repeated work in http_cgi_headers()
save string lengths for reuse
remove trailing slash from strip_request_uri at config time
2021-08-27 02:16:53 -04:00
Glenn Strauss 0286bdef0c [multiple] use thread-safe strerror where avail
use ck_strerror_s() to use strerror_s() or strerror_r() thread-safe and
constraint-checking interfaces, where available.
2021-08-27 02:16:52 -04:00
Glenn Strauss 80848d3708 [core] store int* ptr to common gw status counters
(avoid repeated table lookups)
2021-05-16 21:42:17 -04:00
Glenn Strauss 980554bc70 [core] simplify buffer_path_simplify() 2021-05-08 14:34:05 -04:00
Glenn Strauss 680e6b3bca [multiple] buffer_copy_path_len2() aggregate 2021-04-02 01:16:42 -04:00
Glenn Strauss dc01487ea6 [multiple] use buffer_append_* aggregates
reduces the number of round-trips into some frequently-called routines
2021-04-02 01:16:40 -04:00
Glenn Strauss 19bc88850e [multiple] add attrs from gcc -Wsuggest-attribute= 2021-03-26 07:33:41 -04:00
Glenn Strauss dbe3e2361b [multiple] prefer monotonic time for internal use
Note: monotonic time does not change while VM is suspended

Continue to use real time where required by HTTP protocol, for logging
and for other user-visible instances, such as mod_status, as well as for
external databases and caches.
2021-03-11 18:59:53 -05:00
Glenn Strauss fb5e0ff892 [core] check EAGAIN if unix socket connect() delay 2021-03-08 23:11:53 -05:00
Glenn Strauss db73879bf0 [mod_ajp13] AJPv13 Tomcat connector for lighttpd

AJPv13 protocol reference:
2021-01-29 03:10:22 -05:00
Glenn Strauss 8faa456fc8 [core] fix crash in error trace if backend is down (fixes #3052)
(thx hamelg)

  "segfault with mod_scgi on backend server with unix socket domain"
2021-01-08 17:43:52 -05:00
Glenn Strauss e16b4503e2 [core] fix bug in FastCGI uploads (#3033)
(thx zoon01 and ms49434)

  "Memory Growth with PUT and full buffered streams"
2020-12-26 02:56:39 -05:00
Glenn Strauss 5309ec8c95 [core] fix crash after specific err in config file 2020-12-25 17:47:16 -05:00
Glenn Strauss 1212f60991 buffer_append_path_len() to join paths
use buffer_append_path_len() to join path segments
2020-12-24 16:13:20 -05:00
Glenn Strauss 29f50bdeb0 [multiple] fdevent_waitpid() wrapper 2020-12-24 16:13:17 -05:00
Glenn Strauss ceab1e700d [core] add comment for FastCGI mem use in hctx->rb (#3033)
  "Memory Growth with PUT and full buffered streams"
2020-12-24 01:02:36 -05:00
Glenn Strauss eda12aee22 [core] fix crash printing trace if backend is down 2020-12-16 20:07:21 -05:00
Glenn Strauss 76faed9145 [multiple] replace fall through comment with attr
replace /* fall through */ comment with __attribute_fallthrough__ macro

Note: not adding attribute to code with external origins:
  xxhash.h (algo_xxhash.h)
so to avoid warnings, may need to compile with -Wno-implicit-fallthrough
2020-12-16 05:16:25 -05:00
Glenn Strauss de29a37771 [core] skip some trace if backend Upgrade (#3044)
(thx flynn)

To reduce log noise, skip warning trace reporting error on backend
socket if the connection has been upgraded, e.g. to websockets

  "Socket errors after update to version 1.4.56"
2020-12-09 17:07:43 -05:00
Glenn Strauss f2b33e7520 [multiple] add back-pressure gw data pump (fixes #3033)
When = 0 (the default), the entire request
body is collected before engaging the backend.  For backends which
require data framing, this could lead to growth in memory use as large
requests were framed all at once.

Prefer to retain large request bodies in temporary files on disk and
frame in portions as write queue to backend drains below a threshold.

  "Memory Growth with PUT and full buffered streams"
2020-11-12 17:19:31 -05:00
Glenn Strauss 1efd74457b [core] cold func for gw_recv_response error case
2020-11-04 03:53:15 -05:00
Glenn Strauss 639f70a00f [core] set last_used on rd/wr from backend (fixes #3029)
  "mod_wstunnel kills child on disconnect after idle-timeout from connection start"
2020-11-04 03:53:15 -05:00
Glenn Strauss 81029b8b51 [multiple] inline chunkqueue where always alloc'd
inline struct chunkqueue where always allocated in other structs

(memory locality)
2020-10-11 12:19:27 -04:00
Glenn Strauss 97e314fc9e [multiple] inline chunkqueue_length() 2020-10-11 12:19:26 -04:00