Commit Graph

50 Commits

Author SHA1 Message Date
Glenn Strauss 6be2bd35a1 [core] FILE_CHUNK can hold stat_cache_entry ref 2020-10-20 11:51:48 -04:00
Glenn Strauss 3f1a12e5fb [multiple] remove chunk file.start member
c->offset is now offset into file
c->file.length is end of octets to send (end pos + 1)
c->file.length - c->offset is num of octets to send
2020-10-19 21:40:14 -04:00
Glenn Strauss e18da93e44 [core] reuse chunkqueue_compact_mem*() 2020-10-19 21:40:14 -04:00
Glenn Strauss 81029b8b51 [multiple] inline chunkqueue where always alloc'd
inline struct chunkqueue where always allocated in other structs

(memory locality)
2020-10-11 12:19:27 -04:00
Glenn Strauss 6c68e14940 [core] chunkqueue_{peek,read}_data(), squash
chunkqueue_peek_data(), chunkqueue_read_data(), chunkqueue_read_squash()
shared code for chunkqueue manipulation
2020-10-11 12:19:27 -04:00
Glenn Strauss 97e314fc9e [multiple] inline chunkqueue_length() 2020-10-11 12:19:26 -04:00
Glenn Strauss 9f8a896891 [mod_ssi] update chunk after shell output redirect
chunkqueue_update_file() to update size of file chunk after
redirecting shell output to file fd
2020-10-11 12:19:26 -04:00
Glenn Strauss 5fd8a26a75 [core] defer optimization to read small files
defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured
2020-10-11 12:19:24 -04:00
Glenn Strauss e1bb579361 [core] disperse settings.h to appropriate headers 2020-08-10 20:04:57 -04:00
Glenn Strauss 9914bb297b [core] C99 restrict on some base funcs
buffer.[ch], chunk.[ch], request.[ch], sock_addr.[ch]
2020-07-08 19:54:29 -04:00
Glenn Strauss 1474be7859 [core] move addtl request-specific struct members 2020-07-08 19:54:29 -04:00
Glenn Strauss de73b190a4 [core] read up-to fixed size chunk before fionread 2020-07-08 19:54:28 -04:00
Glenn Strauss 0fcd51438d [core] create http chunk header on the stack
streamline code in http_chunk.c
2020-07-08 19:54:28 -04:00
Glenn Strauss 010c28949c [multiple] prefer (connection *) to (srv *)
convert all log_error_write() to log_error() and pass (log_error_st *)

use con->errh in preference to srv->errh (even though currently same)

avoid passing (server *) when previously used only for logging (errh)
2020-07-08 19:54:28 -04:00
Glenn Strauss ed62e354ff [core] use config_plugin_values_init() 2020-07-08 18:08:51 -04:00
Glenn Strauss 5c0c4936b4 [core] __attribute_returns_nonnull__ 2020-02-24 11:15:32 -05:00
Glenn Strauss 2ad92d17a1 [core] chunkqueue_compact_mem() 2020-02-24 11:14:45 -05:00
Glenn Strauss 72d4fe7451 [core] mark some more funcs w/ __attribute_pure__ 2019-06-06 02:48:43 -04:00
Glenn Strauss f2ac4cdfc5 [core] off_t upload_temp_file_size 2019-03-16 01:39:59 -04:00
Glenn Strauss a1b527e473 [multiple] reduce initial buffer sz if large POST (fixes #2922)
reduce initial buffer size if large POST to backend stored in temp files

regression in lighttpd 1.4.52

(thx rgenoud)

  "[regression] lighttpd gets killed after uploading a big file"
2018-12-12 20:00:07 -05:00
Glenn Strauss 70d7d0a0a1 [core] server.chunkqueue-chunk-sz = 4096
server.chunkqueue-chunk-sz = 4096 (default)
Can be configured any integral value greater than 0.
Value is rounded up to next 1024 if not an even multiple of 1k.

Sites with large request or response headers may benefit from
an 8k or 16k chunk size.  Embedded systems might choose to minimize
memory use by using a 1k chunk size.
2018-11-17 23:53:50 -05:00
Glenn Strauss 88ee73d0a2 [multiple] perf: simplify chunkqueue_get_memory() 2018-11-12 23:01:34 -05:00
Glenn Strauss 1539cbfdc1 [core] perf: chunkqueue buffers already sized up
provide new interface for sizing to even larger if very large request
2018-11-12 21:56:05 -05:00
Glenn Strauss 8c0792a58c [multiple] better packing of struct chunk
remove c->
previous commit modified c-> to share buffer with c->mem
2018-11-12 21:56:05 -05:00
Glenn Strauss fccc7fc607 [core] perf: chunk.c chunk pool
reuse chunks across chunkqueues; remove cq->unused, cq->unused_chunks
reuse same buffer c->mem and c-> in chunk
2018-11-12 08:25:28 -05:00
Glenn Strauss 40d3408328 [core] perf: append response directly into write q
prepend non-dynamic response directly into write queue chunkqueue
2018-10-28 02:38:10 -04:00
Glenn Strauss 23c72fc606 [core] perf: prepend headers directly into write q
prepend response headers directly into write queue chunkqueue
2018-10-27 14:00:08 -04:00
Glenn Strauss f22b5d69da [core] inline chunkqueue_is_empty() 2017-10-09 02:45:12 -04:00
Glenn Strauss ca7a97f211 [core] workaround for AIX mmap define
AIX might #define mmap mmap64 and this conflicts with .mmap member
of struct chunk in chunk.h

  ".mmap in chunk.c on AIX 5.2"
2017-08-08 22:51:32 -04:00
Glenn Strauss b0c66266d9 [core] initialize globals at top of main()
initialize globals (including file-scoped static globals) at top of main
2017-01-31 14:40:05 -05:00
Glenn Strauss 4364a4e029 [core] move network_open_file_chunk() to chunk.c
move network_open_file_chunk() to chunk.c:chunkqueue_open_file_chunk()
for reuse from modules
2017-01-14 01:06:16 -05:00
Glenn Strauss c8e647ad31 [core] set chunkqueue tempdirs at startup
If server.upload-dirs is not configured, then attempt to use TMPDIR
from the environment, if set, or else use /tmp.  Warn at startup if
tempdirs are not present.
2016-07-28 03:57:52 -04:00
Glenn Strauss 1a18669d53 chunkqueue_append_chunkqueue()
(simpler than chunkqueue_steal() for transferring entire chunkqueue)
2016-06-21 15:33:18 -04:00
Glenn Strauss 5a91fd4b90 [core] buffer large responses to tempfiles (fixes #758, fixes #760, fixes #933, fixes #1387, #1283, fixes #2083)
This replaces buffering entire response in memory which might lead to
huge memory footprint and possibly to memory exhaustion.

use tempfiles of fixed size so disk space is freed as each file sent

update callers of http_chunk_append_mem() and http_chunk_append_buffer()
to handle failures when writing to tempfile.

  "memory fragmentation leads to high memory usage after peaks"
  "Random crashing on FreeBSD 6.1"
  "lighty should buffer responses (after it grows above certain size) on disk"
  "Memory usage increases when proxy+ssl+large file"
  "lighttpd+fastcgi memory problem"
  "Excessive Memory usage with streamed files from PHP"
2016-06-12 02:51:10 -04:00
Glenn Strauss 77bd45121c [core] retry tempdirs on partial write, ENOSPC (fixes #2588)
Previous code would fail on partial write, EINTR, and ENOSPC.
Upon any of the above errors, this patch tries next tempdir in list,
if list of tempdirs provided by config option server.upload-dirs

  "Problem when uploading large files"

Closes #54
2016-04-18 04:30:55 -04:00
Glenn Strauss a65c57a548 [core] open fd when appending file to cq (fixes #2655)
http_chunk_append_file() opens fd when appending file to chunkqueue.
Defers calculation of content length until response is finished.

This reduces race conditions pertaining to stat() and then (later)
open(), when the result of the stat() was used for Content-Length
or to generate chunked headers.

Note: this does not change how lighttpd handles files that are modified
in-place by another process after having been opened by lighttpd --
don't do that.  This *does* improve handling of files that are
frequently modified via a temporary file and then atomically renamed
into place.

mod_fastcgi has been modified to use http_chunk_append_file_range() with
X-Sendfile2 and will open the target file multiple times if there are
multiple ranges.

Note: (future todo) not implemented for chunk.[ch] interfaces used by
range requests in mod_staticfile or by mod_ssi.  Those uses could lead
to too many open fds.  For mod_staticfile, limits should be put in place
for max number of ranges accepted by mod_staticfile.  For mod_ssi,
limits would need to be placed on the maximum number of includes, and
the primary SSI file split across lots of SSI directives should either
copy the pieces or perhaps chunk.h could be extended to allow for an
open fd to be shared across multiple chunks.  Doing either of these
would improve the performance of SSI since they would replace many file
opens on the pieces of the SSI file around the SSI directives.

  "Serving a file that is getting updated can cause an empty response or incorrect content-length error"

Closes #49
2016-04-18 04:27:08 -04:00
Glenn Strauss 8abd06a7ff consistent inclusion of config.h at top of files (fixes #2073)
From: Glenn Strauss <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-19 15:14:35 +00:00
Stefan Bühler a069548370 [core] revert increase of temp file size back to 1MB, provide a configure option "server.upload-temp-file-size" instead (fixes #2680)
From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2015-11-07 12:51:14 +00:00
Stefan Bühler 593599f14a rewrite network (write) backends
From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2015-08-22 16:00:59 +00:00
Stefan Bühler 1be163b44a Remove chunkqueue_get_{append,prepend}* API
Although those were "easy" to use, they violated the abstraction:
  content of the chunkqueue should only be modified via the API.
  Replace with chunkqueue_get_memory() and chunkqueue_use_memory() for
  functions that read data from network (reusing large buffers),
  chunkqueue_steal_with_tempfiles() to store request bodies on disk
  Modules that were generating content and need a buffer maintain the
  buffer manually (have to be careful to free the buffer on errors, as
  it isn't part of the chunkqueue yet).

From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2015-02-08 19:10:36 +00:00
Stefan Bühler 6afad87d2e fix buffer, chunk and http_chunk API
* remove unused structs and functions
    (buffer_array, read_buffer)
  * change return type from int to void for many functions,
    as the return value (indicating error/success) was never checked,
    and the function would only fail on programming errors and not on
    invalid input; changed functions to use force_assert instead of
    returning an error.
  * all "len" parameters now are the real size of the memory to be read.
    the length of strings is given always without the terminating 0.
  * the "buffer" struct still counts the terminating 0 in ->used,
    provide buffer_string_length() to get the length of a string in a
    unset config "strings" have used == 0, which is used in some places
    to distinguish unset values from "" (empty string) values.
  * most buffer usages should now use it as string container.
  * optimise some buffer copying by "moving" data to other buffers
  * use (u)intmax_t for generic int-to-string functions
  * remove unused enum values: UNUSED_CHUNK, ENCODING_UNSET
  * converted BUFFER_APPEND_SLASH to inline function (no macro feature
  * refactor: create chunkqueue_steal: moving (partial) chunks into another
  * http_chunk: added separate function to terminate chunked body instead of
    magic handling in http_chunk_append_mem().
    http_chunk_append_* now handle empty chunks, and never terminate the
    chunked body.

From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2015-02-08 12:37:10 +00:00
Stefan Bühler 80552616ce Include mmap.h from chunk.h to fix some problems with #define mmap mmap64 (fixes #1923)
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2009-04-09 17:39:20 +00:00
Marcus Rückert 8cd1471cb3 - white space cleanup part 2 this time 1.4 ;)
i hope it helps with merging stuff back to 1.5

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2006-10-04 13:26:23 +00:00
Jan Kneschke 8517995f52 added append_buffer_weak() which take the buffer directly instead of copying it
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2006-09-21 08:03:40 +00:00
Jan Kneschke 9404220490 chunkqueue can get a list of tempdirs which are tried to open a file at
server.upload-dirs is the option in the configfile

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2005-11-01 07:50:08 +00:00
Jan Kneschke c87a000c13 renamed file.offset to file.start (we had 3 different 'offset' which was confusing)
added a offset tag to the mmap part of the file-struct to allow moving mmap-windows

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2005-10-24 11:36:22 +00:00
Jan Kneschke d958d547c8 added tempfile chunks which remove the file automaticly of they are closed
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2005-09-26 08:53:58 +00:00
Jan Kneschke 63dceeb0c6 open files only once for chunk, cache mmap and let the chunk itself close the fd (fixed #261, #257)
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2005-09-14 08:00:33 +00:00
Jan Kneschke 38b8743af3 - don't keep the full fastcgi response in memory
- don't cache more than 4 chunks in a queue for reuse
- on chunkqueue_reset, clean the chunks

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2005-09-08 10:00:32 +00:00
Jan Kneschke bcdc6a3bbc moved everything below trunk/ and added branches/ and tags/
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2005-02-20 14:27:00 +00:00