This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.
Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.
In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func. In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.
- check for empty strings at config time and set value to NULL if blank
string will be ignored at runtime; at runtime, simple pointer check
for NULL can be used to check for a value that has been set and is not
blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
and use buffer_is_unset() instead of buffer_is_empty(),
where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
truncate string, and use buffer_extend() to extend
Examples where buffer known not to be NULL:
- cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
(though we might set it to NULL if buffer_is_blank(cpv->v.b))
- address of buffer is arg (&foo)
(compiler optimizer detects this in most, but not all, cases)
- buffer is checked for NULL earlier in func
- buffer is accessed in same scope without a NULL check (e.g. b->ptr)
internal behavior change:
callers must not pass a NULL buffer to some funcs.
- buffer_init_buffer() requires non-null args
- buffer_copy_buffer() requires non-null args
- buffer_append_string_buffer() requires non-null args
- buffer_string_space() requires non-null arg
reuse cache lookup in common case of serving a static file
rather than repeating the stat_cache_entry lookup
(which is more work than memcmp() to re-check stat_cache_entry match)
NB: r->tmp_buf == srv->tmp_buf (pointer is copied for quicker access)
NB: request read and write chunkqueues currently point to connection
chunkqueues; per-request and per-connection chunkqueues are
not distinct from one another
con->read_queue == r->read_queue
con->write_queue == r->write_queue
NB: in the future, a separate connection config may be needed for
connection-level module hooks. Similarly, might need to have
per-request chunkqueues separate from per-connection chunkqueues.
Should probably also have a request_reset() which is distinct from
connection_reset().
convert all log_error_write() to log_error() and pass (log_error_st *)
use con->errh in preference to srv->errh (even though currently same)
avoid passing (server *) when previously used only for logging (errh)
More specific checks on contents of array lists. Each module using
lists now does better checking on the types of values in the list
(strings, integers, arrays/lists)
This helps prevent misconfiguration of things like cgi.assign,
fastcgi.server, and scgi.server, where source code might be
served as static files if parenthesis are misplaced.
x-ref:
https://redmine.lighttpd.net/boards/2/topics/6571
move code from mod_staticfile.c to http-header-glue.c to allow reuse
(includes ETag, Last-Modified headers, Range requests for static files)
operate on path arg instead of con->physical.path
skip Range requests if con->http_status already set >= 300
remove redundant calls to stat_cache_get_entry() handling Range requests
x-ref:
"X-Sendfile handoff to mod-static-file in 1.4.x"
https://redmine.lighttpd.net/issues/2017
http_chunk_append_file() opens fd when appending file to chunkqueue.
Defers calculation of content length until response is finished.
This reduces race conditions pertaining to stat() and then (later)
open(), when the result of the stat() was used for Content-Length
or to generate chunked headers.
Note: this does not change how lighttpd handles files that are modified
in-place by another process after having been opened by lighttpd --
don't do that. This *does* improve handling of files that are
frequently modified via a temporary file and then atomically renamed
into place.
mod_fastcgi has been modified to use http_chunk_append_file_range() with
X-Sendfile2 and will open the target file multiple times if there are
multiple ranges.
Note: (future todo) not implemented for chunk.[ch] interfaces used by
range requests in mod_staticfile or by mod_ssi. Those uses could lead
to too many open fds. For mod_staticfile, limits should be put in place
for max number of ranges accepted by mod_staticfile. For mod_ssi,
limits would need to be placed on the maximum number of includes, and
the primary SSI file split across lots of SSI directives should either
copy the pieces or perhaps chunk.h could be extended to allow for an
open fd to be shared across multiple chunks. Doing either of these
would improve the performance of SSI since they would replace many file
opens on the pieces of the SSI file around the SSI directives.
x-ref:
"Serving a file that is getting updated can cause an empty response or incorrect content-length error"
https://redmine.lighttpd.net/issues/2655
github:
Closes #49
- a lot of code tried to handle manually adding terminating zeroes and
keeping track of the correct "used" count.
Replaced all "external" usages with simple wrapper functions:
* buffer_string_is_empty (used <= 1), buffer_is_empty (used == 0);
prefer buffer_string_is_empty
* buffer_string_set_length
* buffer_string_length
* CONST_BUF_LEN() macro
- removed "static" buffer hacks (buffers pointing to constant/stack
memory instead of malloc()ed data)
- buffer_append_strftime(): refactor buffer+strftime uses
- li_tohex(): no need for a buffer for binary-to-hex conversion:
the output data length is easy to predict
- remove "-Winline" from extra warnings: the "inline" keyword just
supresses the warning about unused but defined (static) functions;
don't care whether it actually gets inlined or not.
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2979 152afb58-edef-0310-8abb-c4023f1b3aa9
Although those were "easy" to use, they violated the abstraction:
content of the chunkqueue should only be modified via the API.
Replace with chunkqueue_get_memory() and chunkqueue_use_memory() for
functions that read data from network (reusing large buffers),
chunkqueue_steal_with_tempfiles() to store request bodies on disk
temporarily.
Modules that were generating content and need a buffer maintain the
buffer manually (have to be careful to free the buffer on errors, as
it isn't part of the chunkqueue yet).
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2976 152afb58-edef-0310-8abb-c4023f1b3aa9
* remove unused structs and functions
(buffer_array, read_buffer)
* change return type from int to void for many functions,
as the return value (indicating error/success) was never checked,
and the function would only fail on programming errors and not on
invalid input; changed functions to use force_assert instead of
returning an error.
* all "len" parameters now are the real size of the memory to be read.
the length of strings is given always without the terminating 0.
* the "buffer" struct still counts the terminating 0 in ->used,
provide buffer_string_length() to get the length of a string in a
buffer.
unset config "strings" have used == 0, which is used in some places
to distinguish unset values from "" (empty string) values.
* most buffer usages should now use it as string container.
* optimise some buffer copying by "moving" data to other buffers
* use (u)intmax_t for generic int-to-string functions
* remove unused enum values: UNUSED_CHUNK, ENCODING_UNSET
* converted BUFFER_APPEND_SLASH to inline function (no macro feature
needed)
* refactor: create chunkqueue_steal: moving (partial) chunks into another
queue
* http_chunk: added separate function to terminate chunked body instead of
magic handling in http_chunk_append_mem().
http_chunk_append_* now handle empty chunks, and never terminate the
chunked body.
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2975 152afb58-edef-0310-8abb-c4023f1b3aa9
These should all be non critical:
* memory leaks on startup in error cases (which lead to
immediate shutdowns anyway)
* http_auth/ldap: passing uninitialized "ret" to ldap_err2string
* sizeof(T) not matching the target pointer in malloc/calloc calls;
those cases were either:
* T being the wrong pointer type - shouldn't matter as long as all
pointers have same size
* T being larger than the type needed
* mod_accesslog: direct use after free in cleanup (server shutdown);
could crash before "clean" shutdown
* some false positives (mod_compress, mod_expire)
* assert(srv->config_context->used > 0); - this is always the case,
as there is always a global config block
From: Stefan Bühler <stbuehler@web.de>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2920 152afb58-edef-0310-8abb-c4023f1b3aa9
Fix #1449: Do not add Accept-Ranges header if range-request is disabled
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2090 152afb58-edef-0310-8abb-c4023f1b3aa9
- dont restore the http status code unconditionally (1270)
- mod_staticfile should set the http status properly. otherwise we run into
the 404 handling twice and it appends the default 404 page to the content.
(this passed the testsuite)
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@1899 152afb58-edef-0310-8abb-c4023f1b3aa9
etag.use-size to customize the generation of ETags for
static files. (fixes #1209) (patch by <Yusufg@gmail.com>)
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@1874 152afb58-edef-0310-8abb-c4023f1b3aa9
- this solves the early install problems where the user hasn't setup
fastcgi yet and sends the php-source code out and the browser caches
the file and it's content
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@1725 152afb58-edef-0310-8abb-c4023f1b3aa9
i hope it helps with merging stuff back to 1.5
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@1371 152afb58-edef-0310-8abb-c4023f1b3aa9
Glenn Strauss
Mohammed Sadiq
Stefan Bühler
Jan Kneschke