Commit Graph

4203 Commits (434fab4929abec99683c36bb9ad34aae788bc44e)

Author SHA1 Message Date
Glenn Strauss 434fab4929 [core] quiet coverity warnings 2021-10-01 20:59:08 -04:00
Glenn Strauss 23fe1312e5 [core] chunkqueue_peek_data() mmap experiment
chunkqueue_peek_data() experiment to mmap temporary files

(currently disabled in code due to not making measurable difference
 in throughput (during a specific microbenchmark load test),
 though it does reduce CPU use by ~10% in the same microbenchmark)

enabling this may cause large spikes in RSS mem usage reported by the
system, due to the read-only memory maps of the temporary files,
but this is nothing to be alarmed about, as the memory maps are
file-backed and read-only, so minimally add to memory pressure
2021-10-01 20:32:11 -04:00
Glenn Strauss faf76be187 [core] chunkqueue_mmap_chunk_len() for code reuse
(pull some code into smaller routines for potential code reuse)
2021-10-01 16:11:26 -04:00
Glenn Strauss 6e4cc3b0bb [core] read files into mem when framed for HTTP/2 2021-10-01 06:39:47 -04:00
Glenn Strauss e4cf6998a1 [core] limit initial response header backend read 2021-10-01 06:39:47 -04:00
Glenn Strauss b5d4169156 [core] allow up to 32k of data frames per stream
allow up to 32k of data frames per stream per round
(previously limited to single max_frame_size (default 16k))

For 8 streams, 32k*8 is 256k, which is current lighttpd MAX_WRITE_LIMIT,
so each stream still gets a chance to write data (unless write queue
not emptied on previous attempt, reducing add limit this round)
2021-10-01 06:39:47 -04:00
Glenn Strauss 62dc817caf [core] h2_send_cqdata() returns how much data sent 2021-10-01 06:39:47 -04:00
Glenn Strauss 042622c8c1 [core] use pread() to skip lseek() 2021-10-01 06:39:47 -04:00
Glenn Strauss 60a1224cd0 [core] fix typo in h2_send_cqdata()
fix typo in h2_send_cqdata(); no impact in practice since only caller,
connection_state_machine_h2() already limits dlen to <= max frame size
2021-10-01 06:39:47 -04:00
Glenn Strauss 39c0c2c3ed [core] cap size of data framed for HTTP/2 response
cap size of data framed for HTTP/2 response until more data sent to

make sure to reschedule connection in job queue if max_bytes reached
and then the entire con->write_queue was flushed to network, or else
there is a chance the request may not get rescheduled (and then will
timeout) if the request is completed from the backend and there is
no other traffic or streams to trigger connection processing.

(check con->write_queue > 8k rather than empty from last round,
 since small frames such as connection preface may have been added
 this round while processing con->read_queue)
2021-10-01 06:39:47 -04:00
Glenn Strauss ed2c698300 [tests] ignore SIGINT, SIGUSR1 in fcgi-responder
ignore SIGINT, SIGUSR1 in fcgi-responder if HAVE_SIGNAL is defined
(must be defined separately since config.h is not included)

Not required for test framework.  Added as an example in the code,
e.g. if code is reused with lighttpd and graceful shutdown or restart.
(backend will be sent SIGTERM when server is ready to restart)
2021-10-01 06:39:47 -04:00
Glenn Strauss 41c3cc0965 [core] disable server.graceful-restart-bg if spawn
disable server.graceful-restart-bg if spawning gateway backends
via *.server "bin-path"
2021-10-01 06:39:47 -04:00
Glenn Strauss ff45d5865e [multiple] clarify error msg when no cert avail 2021-10-01 06:39:47 -04:00
Glenn Strauss 6e45cff05a [core] disable streaming response with authorizer (fixes #3106)
disable streaming response while processing "authorizer" mode
until "authorizer" response 200 OK from the backend is complete

(thx jefftharris)

  "FastCGI authorizer hang with"
  "FastCGI authorizer hang with"
2021-10-01 06:39:23 -04:00
Glenn Strauss dd23fcb2b2 [mod_cgi] improve chunk buffer reuse from backends
mark and temporarily defer event handling of backend events in order
to handle at same time as the associated front-end connection events
2021-09-30 17:34:03 -04:00
Glenn Strauss bee35b4877 [core] improve chunk buffer reuse from backends
mark and temporarily defer event handling of backend events in order
to handle at same time as the associated front-end connection events
2021-09-30 17:34:03 -04:00
Glenn Strauss cb7deb493c [core] remove obsolete comment about r->gw_dechunk
r->gw_dechunk->b is not a candidate for using generic chunk buffers.
chunked headers are generally smaller and fit in default 64 byte alloc.
Also, lighttpd limits chunked header to 1k.

Avoid unneeded optimization since HTTP/1.1 use is likely to diminish
over time in favor of HTTP/2 or HTTP/3 or later.
2021-09-30 17:34:03 -04:00
Glenn Strauss 8e6dc4fe94 [core] fix chunked decoding from backend (#3044, #3046)
fix edge case for initial chunked data
(bug introduced in lighttpd 1.4.56)

If chunked header received without data before response headers sent,
then initial chunked data might be sent to client without chunked header
if client made an HTTP/1.1 request and the response is Transfer-Encoding
chunked and lighttpd is configured to stream the response (non-zero
value for  This might occur if lighttpd
backend is connected via a unix domain socket and the initial chunk is
large and coming from a temporary file.  It may be sent in a separate
packet since lighttpd does not use TCP_CORK on unix domain sockets.

  "Failure on second request in http proxy backend"
  "Socket errors after update to version 1.4.56"
2021-09-30 17:34:03 -04:00
Glenn Strauss 6e62b84258 [core] splice() data from backends to tempfiles
splice() data from backends to tempfiles (where splice() is available);
reduce copying data to userspace when writing data to tempfiles

Note: splice() on Linux returns EINVAL if target file has O_APPEND set
so lighttpd uses pwrite() (where available) when writing to tempfiles
(instead of lseek() + write(), or O_APPEND and write())
2021-09-30 17:34:03 -04:00
Glenn Strauss 6bd6226e90 [multiple] avoid empty chunks in chunkqueue
(This might render some calls to chunkqueue_remove_finished_chunks()
 no longer necessary)
2021-09-30 17:34:03 -04:00
Glenn Strauss 4d99d9b78a [multiple] check feature flags funcs; code reuse
2021-09-30 17:34:03 -04:00
Glenn Strauss f37847b1f6 [core] _WIN32 impl of fdevent_mkostemp() 2021-09-30 17:34:03 -04:00
Glenn Strauss cc2fcd3ece [multiple] _WIN32 fdevent_pipe_cloexec()
Note: Under _WIN32, serious limitation in Windows APIs:
  select() and WSAPoll() operate only on sockets (not pipes)
  (directly affects mod_cgi; not currently handled)
2021-09-30 17:34:03 -04:00
Glenn Strauss 5cab11b7ea [build] check for pread(), pwrite(), splice() 2021-09-30 17:34:03 -04:00
Glenn Strauss 9f62f1b196 [multiple] fdevent_mkostemp()
fdevent_mkostemp() with flags arg so that caller can pass O_APPEND

renamed from fdevent_mkstemp_append(), previously always O_APPEND
2021-09-30 17:34:03 -04:00
Glenn Strauss 2e60c4e0b2 [core] remove redundant checks for tempfile chunk
remove redundant checks for tempfile chunk reuse
c->file.is_temp is only set if c->type == FILE_CHUNK is also true

The test for (0 == c->offset) is historical.  Before the temporary files
were opened O_APPEND (or written to using pwrite()), the file offset may
have changed via lseek() if lighttpd had started reading the file to
send to the client.  To avoid this, the (0 == c->offset) check was used
as a quick check to avoid continuing to write to a temporary file that
lighttpd had begun to read.
2021-09-30 17:34:03 -04:00
Glenn Strauss d530a49d38 [core] chunkqueue_get_append_newtempfile()
rename chunkqueue_get_append_tempfile()
    -> chunkqueue_get_append_newtempfile()

pull some code from chunkqueue_append_mem_to_tempfile()
  into smaller func for (new func) chunkqueue_get_append_tempfile(),
  which might call into chunkqueue_get_append_newtempfile()

pull some code from chunkqueue_append_mem_to_tempfile()
  into smaller func chunkqueue_append_tempfile_err()
  to handle write errors with respect to removing empty chunk
  and stepping to next configured tempdir
2021-09-30 17:34:03 -04:00
Glenn Strauss b9f143be69 [core] create temp file name in chunk buffer
create temp file name directly in chunk buffer
rather than using a separate allocation
2021-09-30 17:34:03 -04:00
Glenn Strauss 0b6de09071 [core] option: errorlog high precision timestamps
server.feature-flags += ("server.errorlog-high-precision" => "enable")

Note: if using syslog() for errorlog, modern syslog implementations are
      configured separately (by an admin) for high precision timestamps;
      server.feature-flags has no effect on syslog-generated timestamps

restructure some of log.c into smaller internal routines

keep a file-scoped global log_stderrh to write to STDERR_FILENO
so that an errh handle is always available for logging errors

fix missing space between timestamp and filename in errorlog output
(missing space in lighttpd 1.4.58 and lighttpd 1.4.59) (fixes #3105)
  "missing ( in log lines from mod_auth"
2021-09-30 17:33:01 -04:00
Glenn Strauss f19f71625c [multiple] internal control for backend read bytes
separate internal control for backend max_per_read

When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a smaller buffer to help reduce memory usage.

When not streaming, attempt to read and empty kernel socket bufs.
  (e.g. MAX_READ_LIMIT 256k)

When writing to sockets (or pipes) attempt to fill kernel socket bufs.
  (e.g. MAX_WRITE_LIMIT 256k)
2021-09-28 11:05:55 -04:00
Glenn Strauss d59d5e59b9 [core] improve chunk buffer reuse from backends
improve oversized chunk buffer reuse from backends
2021-09-24 10:44:50 -04:00
Glenn Strauss 8d63c7900d [core] retain largest chunk on oversized chunk lst
retain largest chunk on oversized chunk list if list size limit reached
2021-09-17 19:17:37 -04:00
Glenn Strauss b18e9dbcab [core] separate mem pool for FILE_CHUNK reuse
file names tend to be much shorter than chunk_buf_sz
so using separate pool saves memory for large request and
response bodies where many temporary files are collected
2021-09-17 19:17:37 -04:00
Glenn Strauss b38817b678 [core] mark attr malloc, returns nonnull
mark some core code with attributes malloc, returns nonnull
2021-09-17 05:06:16 -04:00
Glenn Strauss dc2d82811c [core] HTTP/2 GOAWAY after timeout before read (#3102)
HTTP/2 send GOAWAY soon after client timeout, before potentially
reading new stream requests, which will then have to be reset.

  "Chrome gives random net::ERR_HTTP2_PROTOCOL_ERROR"
2021-09-16 14:43:51 -04:00
Glenn Strauss 6ffabc9607 [core] default backend "connect-timeout" to 8 (#3086)
default backend "connect-timeout" to 8 seconds

Though this is is a behavior change where there previously was no
timeout, this is configurable by lighttpd.conf admin, and having a
default connection timeout of a fairly large value (8 seconds) puts
a (default) limit on resource usage waiting for socket connect().

  "sockets disabled, out-of-fds with proxy module"
2021-09-16 04:40:13 -04:00
Glenn Strauss 94bac825dc [core] HTTP/2 GOAWAY after timeout before read (fixes #3102)
HTTP/2 send GOAWAY soon after keep-alive timeout, before potentially
reading new stream requests, which will then have to be reset.

  "Chrome gives random net::ERR_HTTP2_PROTOCOL_ERROR"
2021-09-16 04:40:13 -04:00
Glenn Strauss 0b56c16a8b [core] reduce oversized mem alloc for backends
reduce oversized memory allocations when reading from backends:
avoid extra power-2 allocation for 1 byte ('\0') when data
available to read is exactly power-2
2021-09-16 04:40:12 -04:00
Glenn Strauss 94053349c2 [core] remove redundant check in h2_recv_headers()
(comment out redundant check in h2_recv_headers())
2021-09-15 23:40:26 -04:00
Glenn Strauss a407636eae [core] reject HTTP/2 pseudo-header in trailers (#3102)
(detect if client erroneously reuses stream id for a different request)

  "Chrome gives random net::ERR_HTTP2_PROTOCOL_ERROR"
2021-09-15 05:11:54 -04:00
Glenn Strauss af96e52be9 [core] quiet coverity warnings 2021-09-14 03:25:58 -04:00
Glenn Strauss 3a5704380b [core] refuse excess initial streams only if block (fixes #3100)
refuse excess streams only if would block DATA frames for active streams
(for excess streams received on initial connect, prior to receiving
SETTINGS ACK from client)

(thx flynn)

  "Random TLS errors on established connections"
  "Chrome 92, HTTP/2, fcgi, mutiple puts no response"
2021-09-14 00:05:25 -04:00
Glenn Strauss 58b9cca536 [core] shared temp buffer for log_*error*()
use shared temp buffer for preparing error log entries
(each error log entry is flushed to error log;
 there is no persistent data buffering for error logs)
2021-09-13 03:37:40 -04:00
Glenn Strauss 062ea98bcb [multiple] prefer r->tmp_buf to p->tmp_buf
prefer per-request r->tmp_buf to per-module p->tmp_buf
to marginally increase buf reuse during each request.
(currently, r->tmp_buf == srv->tmp_buf)

(avoid some persistent memory allocations per-module,
 as those are not currently cleared/released periodically)
2021-09-13 03:37:40 -04:00
Glenn Strauss 7b615d5d24 [multiple] de-dup file and piped loggers (fixes #3101)
de-dup file and piped loggers for error logs and access logs

  "RFE: de-dup file and piped loggers"
2021-09-13 03:37:11 -04:00
Glenn Strauss 243510dbb4 [core] fdlog.[ch]; fdevent_*_logger_* -> fdlog_*
code move and rename fdevent_*_logger_*() to fdlog_*()
2021-09-11 05:31:09 -04:00
Glenn Strauss 4763e08c07 [core] update comment about server.max-write-idle 2021-09-09 16:14:33 -04:00
Glenn Strauss e97a5b7e49 [core] clear buffer after backend dechunk
(thx flynn)

clear buffer after backend dechunk if not sending chunked to client

  "Memory fragmentation with HTTP/2 enabled"
2021-09-09 16:12:37 -04:00
Glenn Strauss 9a37ba54d1 [tests] skip time-sensitive test during CI testing 2021-09-09 04:55:26 -04:00
Glenn Strauss f99cb7d7ab [core] quiet coverity warnings 2021-09-09 04:08:26 -04:00