Commit Graph

4283 Commits (be82f3d2e1e65faa54d335b53458294ef7ec8e8c)

Author SHA1 Message Date
Glenn Strauss 4d99d9b78a [multiple] check feature flags funcs; code reuse
2021-09-30 17:34:03 -04:00
Glenn Strauss f37847b1f6 [core] _WIN32 impl of fdevent_mkostemp() 2021-09-30 17:34:03 -04:00
Glenn Strauss cc2fcd3ece [multiple] _WIN32 fdevent_pipe_cloexec()
Note: Under _WIN32, serious limitation in Windows APIs:
  select() and WSAPoll() operate only on sockets (not pipes)
  (directly affects mod_cgi; not currently handled)
2021-09-30 17:34:03 -04:00
Glenn Strauss 5cab11b7ea [build] check for pread(), pwrite(), splice() 2021-09-30 17:34:03 -04:00
Glenn Strauss 9f62f1b196 [multiple] fdevent_mkostemp()
fdevent_mkostemp() with flags arg so that caller can pass O_APPEND

renamed from fdevent_mkstemp_append(), previously always O_APPEND
2021-09-30 17:34:03 -04:00
Glenn Strauss 2e60c4e0b2 [core] remove redundant checks for tempfile chunk
remove redundant checks for tempfile chunk reuse
c->file.is_temp is only set if c->type == FILE_CHUNK is also true

The test for (0 == c->offset) is historical.  Before the temporary files
were opened O_APPEND (or written to using pwrite()), the file offset may
have changed via lseek() if lighttpd had started reading the file to
send to the client.  To avoid this, the (0 == c->offset) check was used
as a quick check to avoid continuing to write to a temporary file that
lighttpd had begun to read.
2021-09-30 17:34:03 -04:00
Glenn Strauss d530a49d38 [core] chunkqueue_get_append_newtempfile()
rename chunkqueue_get_append_tempfile()
    -> chunkqueue_get_append_newtempfile()

pull some code from chunkqueue_append_mem_to_tempfile()
  into smaller func for (new func) chunkqueue_get_append_tempfile(),
  which might call into chunkqueue_get_append_newtempfile()

pull some code from chunkqueue_append_mem_to_tempfile()
  into smaller func chunkqueue_append_tempfile_err()
  to handle write errors with respect to removing empty chunk
  and stepping to next configured tempdir
2021-09-30 17:34:03 -04:00
Glenn Strauss b9f143be69 [core] create temp file name in chunk buffer
create temp file name directly in chunk buffer
rather than using a separate allocation
2021-09-30 17:34:03 -04:00
Glenn Strauss 0b6de09071 [core] option: errorlog high precision timestamps
server.feature-flags += ("server.errorlog-high-precision" => "enable")

Note: if using syslog() for errorlog, modern syslog implementations are
      configured separately (by an admin) for high precision timestamps;
      server.feature-flags has no effect on syslog-generated timestamps

restructure some of log.c into smaller internal routines

keep a file-scoped global log_stderrh to write to STDERR_FILENO
so that an errh handle is always available for logging errors

fix missing space between timestamp and filename in errorlog output
(missing space in lighttpd 1.4.58 and lighttpd 1.4.59) (fixes #3105)
  "missing ( in log lines from mod_auth"
2021-09-30 17:33:01 -04:00
Glenn Strauss f19f71625c [multiple] internal control for backend read bytes
separate internal control for backend max_per_read

When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a smaller buffer to help reduce memory usage.

When not streaming, attempt to read and empty kernel socket bufs.
  (e.g. MAX_READ_LIMIT 256k)

When writing to sockets (or pipes) attempt to fill kernel socket bufs.
  (e.g. MAX_WRITE_LIMIT 256k)
2021-09-28 11:05:55 -04:00
Glenn Strauss d59d5e59b9 [core] improve chunk buffer reuse from backends
improve oversized chunk buffer reuse from backends
2021-09-24 10:44:50 -04:00
Glenn Strauss 8d63c7900d [core] retain largest chunk on oversized chunk lst
retain largest chunk on oversized chunk list if list size limit reached
2021-09-17 19:17:37 -04:00
Glenn Strauss b18e9dbcab [core] separate mem pool for FILE_CHUNK reuse
file names tend to be much shorter than chunk_buf_sz
so using separate pool saves memory for large request and
response bodies where many temporary files are collected
2021-09-17 19:17:37 -04:00
Glenn Strauss b38817b678 [core] mark attr malloc, returns nonnull
mark some core code with attributes malloc, returns nonnull
2021-09-17 05:06:16 -04:00
Glenn Strauss dc2d82811c [core] HTTP/2 GOAWAY after timeout before read (#3102)
HTTP/2 send GOAWAY soon after client timeout, before potentially
reading new stream requests, which will then have to be reset.

  "Chrome gives random net::ERR_HTTP2_PROTOCOL_ERROR"
2021-09-16 14:43:51 -04:00
Glenn Strauss 6ffabc9607 [core] default backend "connect-timeout" to 8 (#3086)
default backend "connect-timeout" to 8 seconds

Though this is is a behavior change where there previously was no
timeout, this is configurable by lighttpd.conf admin, and having a
default connection timeout of a fairly large value (8 seconds) puts
a (default) limit on resource usage waiting for socket connect().

  "sockets disabled, out-of-fds with proxy module"
2021-09-16 04:40:13 -04:00
Glenn Strauss 94bac825dc [core] HTTP/2 GOAWAY after timeout before read (fixes #3102)
HTTP/2 send GOAWAY soon after keep-alive timeout, before potentially
reading new stream requests, which will then have to be reset.

  "Chrome gives random net::ERR_HTTP2_PROTOCOL_ERROR"
2021-09-16 04:40:13 -04:00
Glenn Strauss 0b56c16a8b [core] reduce oversized mem alloc for backends
reduce oversized memory allocations when reading from backends:
avoid extra power-2 allocation for 1 byte ('\0') when data
available to read is exactly power-2
2021-09-16 04:40:12 -04:00
Glenn Strauss 94053349c2 [core] remove redundant check in h2_recv_headers()
(comment out redundant check in h2_recv_headers())
2021-09-15 23:40:26 -04:00
Glenn Strauss a407636eae [core] reject HTTP/2 pseudo-header in trailers (#3102)
(detect if client erroneously reuses stream id for a different request)

  "Chrome gives random net::ERR_HTTP2_PROTOCOL_ERROR"
2021-09-15 05:11:54 -04:00
Glenn Strauss af96e52be9 [core] quiet coverity warnings 2021-09-14 03:25:58 -04:00
Glenn Strauss 3a5704380b [core] refuse excess initial streams only if block (fixes #3100)
refuse excess streams only if would block DATA frames for active streams
(for excess streams received on initial connect, prior to receiving
SETTINGS ACK from client)

(thx flynn)

  "Random TLS errors on established connections"
  "Chrome 92, HTTP/2, fcgi, mutiple puts no response"
2021-09-14 00:05:25 -04:00
Glenn Strauss 58b9cca536 [core] shared temp buffer for log_*error*()
use shared temp buffer for preparing error log entries
(each error log entry is flushed to error log;
 there is no persistent data buffering for error logs)
2021-09-13 03:37:40 -04:00
Glenn Strauss 062ea98bcb [multiple] prefer r->tmp_buf to p->tmp_buf
prefer per-request r->tmp_buf to per-module p->tmp_buf
to marginally increase buf reuse during each request.
(currently, r->tmp_buf == srv->tmp_buf)

(avoid some persistent memory allocations per-module,
 as those are not currently cleared/released periodically)
2021-09-13 03:37:40 -04:00
Glenn Strauss 7b615d5d24 [multiple] de-dup file and piped loggers (fixes #3101)
de-dup file and piped loggers for error logs and access logs

  "RFE: de-dup file and piped loggers"
2021-09-13 03:37:11 -04:00
Glenn Strauss 243510dbb4 [core] fdlog.[ch]; fdevent_*_logger_* -> fdlog_*
code move and rename fdevent_*_logger_*() to fdlog_*()
2021-09-11 05:31:09 -04:00
Glenn Strauss 4763e08c07 [core] update comment about server.max-write-idle 2021-09-09 16:14:33 -04:00
Glenn Strauss e97a5b7e49 [core] clear buffer after backend dechunk
(thx flynn)

clear buffer after backend dechunk if not sending chunked to client

  "Memory fragmentation with HTTP/2 enabled"
2021-09-09 16:12:37 -04:00
Glenn Strauss 9a37ba54d1 [tests] skip time-sensitive test during CI testing 2021-09-09 04:55:26 -04:00
Glenn Strauss f99cb7d7ab [core] quiet coverity warnings 2021-09-09 04:08:26 -04:00
Glenn Strauss f9c817680d [build] define rsize_t on FreeBSD
(unsure why lighttpd CI CMake build has rsize_t defined,
 but not autotools or SCons lighttpd CI builds)
2021-09-09 03:34:55 -04:00
Glenn Strauss a3e9faa479 [multiple] quiet coverity warnings 2021-09-09 02:16:21 -04:00
Glenn Strauss f364c8ef36 [core] remove redundant waitpid() on each backend
remove redundant waitpid() on each backend
(performed each second during periodic maintenance)

Doing so is redundant with handle_waitpid hook since lighttpd 1.4.46
2021-09-08 15:06:07 -04:00
Glenn Strauss e77bc3995c [core] reduce wait time in gw_spawn_connection()
reduce wait time in gw_spawn_connection() from 10ms down to 1ms.
That is still plenty of time to detect immediate execve()/clone()
2021-09-08 15:06:07 -04:00
Glenn Strauss 90978eb808 [core] mark cold more gateway maintenance code 2021-09-08 15:06:07 -04:00
Glenn Strauss 9c7ef4f064 [core] consolidate more gw_host, gw_proc init code
clearer code: host->num_procs as num gw_proc in host->first linked list
2021-09-08 15:06:07 -04:00
Glenn Strauss 33ffec35ad [mod_cgi] cgi.limits "tcp-fin-propagate" => "SIG"
cgi.limits += ("tcp-fin-propagate" => "<signal>")  (e.g. "SIGTERM")

send specified signal to CGI if TCP FIN is received from client
(default: do not send signal to CGI)
2021-09-08 15:06:07 -04:00
Glenn Strauss ee9352b1bb [core] realloc buffer power-2 size + 1 for '\0'
realloc buffer power-2 size + 1 for '\0' to avoid power-2 allocation
doubling buffer size for sole reason of storing '\0' at end of block
2021-09-08 15:06:07 -04:00
Glenn Strauss 715699ba50 [core] write all cq MEM_CHUNK if spill to tempfile
If chunkqueue size grows large enough in memory to use tempfiles,
write all MEM_CHUNK in chunkqueue to tempfiles to free up memory.

If earlier chunks in chunkqueue are MEM_CHUNK, then a prior attempt
to write was not able to proceed, or items are being added to
chunkqueue as they are decoded from TLS.  In either case, choose to
free up memory sooner.  This may result in some cases where lighttpd
soon reads data back from disk to send to the backend, but since some
(now all) of the data is in tempfiles, sendfile() will avoid reading
back into userspace, though sendfile() will not be used if lighttpd
needs to read data back into memory to encrypt data using TLS.
2021-09-08 15:06:07 -04:00
Glenn Strauss 6800b0821c [core] reduce stack use in main()
mark noinline for setup and other cold funcs
2021-09-08 15:06:07 -04:00
Glenn Strauss ec88f62b5a [core] change backtrace format to put addr first
(better monospaced alignment of frame num, addr, offset, name)
2021-09-08 15:06:07 -04:00
Glenn Strauss 84ff9ac9c9 [core] expose ck_bt() for debugging 2021-09-08 15:06:07 -04:00
Glenn Strauss bd72d1927a [core] remove server.upload-temp-file-size limit
previously undocumented server.upload-temp-file-size in lighttpd 1.4.38
preceded introduction of lighttpd streaming options in lighttpd 1.4.40
( and
2021-09-08 15:06:07 -04:00
Glenn Strauss 33808a2e73 [core] TCP_CORK w/ MEM_CHUNK then FILE_CHUNK
or multiple MEM_CHUNK with TLS
2021-09-08 15:06:07 -04:00
Glenn Strauss ad8a27f356 [core] cfg server.bindhost after $SERVER["socket"]
init global config for server.bindhost and server.port after
initializing $SERVER["socket"] so that if bindhost and port matches
another $SERVER["socket"], the $SERVER["socket"] config is used,
as the $SERVER["socket"] config inherits from the global scope and
can the be overridden.

  "Activate SSL with lighttpd on a Raspberry Pi"
2021-09-08 15:06:07 -04:00
Glenn Strauss cd738d4daa [build] propagate HAVE_DLFCN_H in builds 2021-09-08 15:06:07 -04:00
Glenn Strauss 1a8d8e0ca5 [core] periodically malloc_trim() to release mem (fixes #3084)
  "Memory fragmentation with HTTP/2 enabled"
2021-09-08 15:06:07 -04:00
Glenn Strauss 43f0106f81 [core] config mallopt(M_ARENA_MAX, 2) (#3084)
Preemptively cap the max number of arenas that might be created by glibc

Each thread attempts to use the previously-used arena.
lighttpd is single-threaded, so in general, only one arena is used.

  "Memory fragmentation with HTTP/2 enabled"
2021-09-08 15:06:07 -04:00
Glenn Strauss 1783550154 [build] look for malloc.h and mallopt() 2021-09-08 15:06:07 -04:00
Glenn Strauss 18ec4f419c [tests] move tests/docroot/www contents up 1 level
reduce directories under tests
(because automake is sloooow and the fewer dirs, the better)
2021-09-08 15:06:07 -04:00