Commit Graph

99 Commits

Author SHA1 Message Date
Glenn Strauss 37ae942346 [core] fix bug in read retry found by coverity
read retry loop needs separate var for result and data size to read

rename 'toSend' variable to 'len' (of chunk)
2020-12-27 22:35:48 -05:00
Glenn Strauss 2639e5ae43 [multiple] chunkqueue_write_chunk()
create API in chunk.[ch] for writing a chunk to an fd

(pull similar code from mod_cgi and mod_webdav)

This new API is intended for use on request body input, which is
written to size-limited temporary files controlled by lighttpd and
written to files or pipes.

(network_backend_write() is for writing chunkqueues to sockets)
2020-12-27 09:17:33 -05:00
Glenn Strauss 7ba521ffb4 [core] reuse large mem chunks (fix mem usage) (fixes #3033)
(thx flynn)

fix large memory usage for large file downloads from dynamic backends

reuse or release large memory chunks

  "Memory Growth with PUT and full buffered streams"
2020-12-24 00:20:27 -05:00
Glenn Strauss 1d9cab1bc3 [core] always set file.fd = -1 on FILE_CHUNK reset (fixes #3044)
(thx flynn)

  "Socket errors after update to version 1.4.56"
2020-12-09 17:07:37 -05:00
Glenn Strauss c42b280583 [core] fix bug when HTTP/2 frames span chunks
fix inverted logic when HTTP/2 frames span chunkqueue chunks
2020-11-13 17:00:01 -05:00
Glenn Strauss 8f7019adf0 [core] fix fd sharing when splitting file chunk
(bug on master branch; never released)

fix fd sharing in chunkqueue_steal_partial_file_chunk()
2020-10-20 16:53:51 -04:00
Glenn Strauss 1b74c50854 [core] always lseek() with shared fd
always lseek() with shared fd; remove optim to skip with offset = 0
2020-10-20 11:51:48 -04:00
Glenn Strauss 6be2bd35a1 [core] FILE_CHUNK can hold stat_cache_entry ref 2020-10-20 11:51:48 -04:00
Glenn Strauss 3f1a12e5fb [multiple] remove chunk file.start member
c->offset is now offset into file
c->file.length is end of octets to send (end pos + 1)
c->file.length - c->offset is num of octets to send
2020-10-19 21:40:14 -04:00
Glenn Strauss e18da93e44 [core] reuse chunkqueue_compact_mem*() 2020-10-19 21:40:14 -04:00
Glenn Strauss 6d76f7ce49 [core] fix chunkqueue_compact_mem w/ partial chunk
(bug on master branch; never released)
2020-10-19 21:40:14 -04:00
Glenn Strauss 14eb97f1b3 [core] dup FILE_CHUNK fd when splitting FILE_CHUNK 2020-10-11 12:19:27 -04:00
Glenn Strauss 7c1e81299f [core] handle unexpected EOF reading FILE_CHUNK
(replace existing check which suffered from ToC-ToU race condition)
enhances logic from 2015 commit 593599f1 and avoids repeated fstat()
checks when sending large files

For mmap(), lighttpd catches SIGBUS if file is (externally) truncated
and lighttpd attempts to access bytes in a read-only mapping more than
a memory page boundary following the end of the file.

For sendfile(), lighttpd returns an error if sendfile() reports no error
and that no bytes have been sent after lighttpd attempts to send a
non-zero number of bytes.
2020-10-11 12:19:27 -04:00
Glenn Strauss 81029b8b51 [multiple] inline chunkqueue where always alloc'd
inline struct chunkqueue where always allocated in other structs

(memory locality)
2020-10-11 12:19:27 -04:00
Glenn Strauss 6c68e14940 [core] chunkqueue_{peek,read}_data(), squash
chunkqueue_peek_data(), chunkqueue_read_data(), chunkqueue_read_squash()
shared code for chunkqueue manipulation
2020-10-11 12:19:27 -04:00
Glenn Strauss 97e314fc9e [multiple] inline chunkqueue_length() 2020-10-11 12:19:26 -04:00
Glenn Strauss 9f8a896891 [mod_ssi] update chunk after shell output redirect
chunkqueue_update_file() to update size of file chunk after
redirecting shell output to file fd
2020-10-11 12:19:26 -04:00
Glenn Strauss 5fd8a26a75 [core] defer optimization to read small files
defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured
2020-10-11 12:19:24 -04:00
Glenn Strauss d2d5f27f80 [core] chunkqueue_compact_mem() must upd cq->last 2020-08-10 12:54:33 -04:00
Glenn Strauss 1d6003e575 [core] mark some chunk.c funcs as pure 2020-07-08 19:54:29 -04:00
Glenn Strauss 9914bb297b [core] C99 restrict on some base funcs
buffer.[ch], chunk.[ch], request.[ch], sock_addr.[ch]
2020-07-08 19:54:29 -04:00
Glenn Strauss 304e46d4f8 [core] default chunk size 8k (was 4k) 2020-07-08 19:54:28 -04:00
Glenn Strauss de73b190a4 [core] read up-to fixed size chunk before fionread 2020-07-08 19:54:28 -04:00
Glenn Strauss 0fcd51438d [core] create http chunk header on the stack
streamline code in http_chunk.c
2020-07-08 19:54:28 -04:00
Glenn Strauss 010c28949c [multiple] prefer (connection *) to (srv *)
convert all log_error_write() to log_error() and pass (log_error_st *)

use con->errh in preference to srv->errh (even though currently same)

avoid passing (server *) when previously used only for logging (errh)
2020-07-08 19:54:28 -04:00
Glenn Strauss ed62e354ff [core] use config_plugin_values_init() 2020-07-08 18:08:51 -04:00
Glenn Strauss 601c572c39 [core] inline buffer as part of data_string value
(instead of value being (buffer *))
2020-02-24 11:15:32 -05:00
Glenn Strauss 5c0c4936b4 [core] __attribute_returns_nonnull__ 2020-02-24 11:15:32 -05:00
Glenn Strauss 2ad92d17a1 [core] chunkqueue_compact_mem() 2020-02-24 11:14:45 -05:00
Glenn Strauss d9e2dd4b2e [core] chunkqueue perf: specialized buffer.h funcs 2019-05-13 20:27:49 -04:00
Glenn Strauss 6572976e06 [core] chunkqueue perf: code reuse
code reuse, simplification, and inlining
remove excess calls to chunkqueue_remove_finished_chunks()

(it may still be possible for there to be an empty chunk in chunkqueue
 if nothing were written to a temporary file (need to verify this), so
 preserve some calls to chunkqueue_remove_finished_chunks() for now)
2019-05-13 20:27:49 -04:00
Glenn Strauss 599b4f05c8 [core] fix 1.4.52 regression in mem use with POST (fixes #2948)
(thx rgenoud)

  "[regression][Bisected] lighttpd uses way more memory with POST since 1.4.52"
2019-04-20 02:06:43 -04:00
Glenn Strauss f2ac4cdfc5 [core] off_t upload_temp_file_size 2019-03-16 01:39:59 -04:00
Glenn Strauss a78404cfbf [core] fdevent_mkstemp_append() (shared) 2019-03-16 01:38:29 -04:00
Glenn Strauss 37bd124ae4 [core] pass conf.follow_symlink in more places 2019-03-10 23:22:58 -04:00
Glenn Strauss c83fff1dda [core] release empty chunk buf when nothing read 2019-03-07 00:32:17 -05:00
Glenn Strauss a1b527e473 [multiple] reduce initial buffer sz if large POST (fixes #2922)
reduce initial buffer size if large POST to backend stored in temp files

regression in lighttpd 1.4.52

(thx rgenoud)

  "[regression] lighttpd gets killed after uploading a big file"
2018-12-12 20:00:07 -05:00
Glenn Strauss 77c01f9817 [core] buffer_append_path_len()
concatenate paths, placing single '/' between strings

reverts broken commit:b9402283

(thx avij)
2018-11-28 02:12:41 -05:00
Glenn Strauss f69bd9cdb8 [core] perf: simple, quick buffer_clear()
quickly clear buffer instead of buffer_string_set_length(b, 0) or
buffer_reset(b).  Avoids free() of large buffers about to be reused,
or buffers that are module-scoped, persistent, and reused.

(buffer_reset() should still be used with buffers in connection *con
 when the data in the buffers is supplied by external, untrusted source)
2018-11-23 00:37:38 -05:00
Glenn Strauss 9749503b1d [core] separate func to reset FILE_CHUNK 2018-11-21 00:23:40 -05:00
Glenn Strauss 758c58cda5 [core] clear chunk buffer upon release
(before adding back to reusable buffer pool)
2018-11-18 17:02:06 -05:00
Glenn Strauss 70d7d0a0a1 [core] server.chunkqueue-chunk-sz = 4096
server.chunkqueue-chunk-sz = 4096 (default)
Can be configured any integral value greater than 0.
Value is rounded up to next 1024 if not an even multiple of 1k.

Sites with large request or response headers may benefit from
an 8k or 16k chunk size.  Embedded systems might choose to minimize
memory use by using a 1k chunk size.
2018-11-17 23:53:50 -05:00
Glenn Strauss 88ee73d0a2 [multiple] perf: simplify chunkqueue_get_memory() 2018-11-12 23:01:34 -05:00
Glenn Strauss f512bfa4af [core] use chunk_buf_sz instead of hard-coded num 2018-11-12 21:56:05 -05:00
Glenn Strauss 1539cbfdc1 [core] perf: chunkqueue buffers already sized up
provide new interface for sizing to even larger if very large request
2018-11-12 21:56:05 -05:00
Glenn Strauss 8c0792a58c [multiple] better packing of struct chunk
remove c->
previous commit modified c-> to share buffer with c->mem
2018-11-12 21:56:05 -05:00
Glenn Strauss fccc7fc607 [core] perf: chunk.c chunk pool
reuse chunks across chunkqueues; remove cq->unused, cq->unused_chunks
reuse same buffer c->mem and c-> in chunk
2018-11-12 08:25:28 -05:00
Glenn Strauss a9ae35128d [core] chunk.c code reuse 2018-10-28 10:00:03 -04:00
Glenn Strauss 40d3408328 [core] perf: append response directly into write q
prepend non-dynamic response directly into write queue chunkqueue
2018-10-28 02:38:10 -04:00
Glenn Strauss 59a71815dc [core] perf: copy small strings; extend last chunk
copy small strings; extend last chunk
(instead of creating new, tiny chunk)
2018-10-28 02:38:06 -04:00