Commit Graph

75 Commits

Author SHA1 Message Date
Glenn Strauss 48384c7e2f [core] fix (startup) mem leaks in configparser.y
(thx stbuehler)
2020-10-24 17:06:50 -04:00
Glenn Strauss 2e0676fd6d [core] extend (data_string *) to store header id
(optional addition to (data_string *), used by http_header.[ch])

extend (data_string *) instead of creating another data_* TYPE_*
  (new data type would probably have (data_string *) as base class)
  (might revisit choice in the future)

HTTP_HEADER_UNSPECIFIED has been removed.  It was used in select
locations as an optimization to avoid looking up enum header_header_e
before checking the array, but the ordering in the array now relies
on having the id.  Having the id allows for a quick check if a common
header is present or not in the htags bitmask, before checking the
array, and allows for integer comparison in the log(n) search of the
array, instead of strncasecmp().

With HTTP_HEADER_UNSPECIFIED removed, add optimization to set bit
in htags for HTTP_HEADER_OTHER when an "other" header is added,
but do not clear the bit, as there might be addtl "other" headers
2020-10-11 12:19:26 -04:00
Glenn Strauss 446866333f [multiple] use *(unsigned char *) with ctypes 2020-07-08 22:51:31 -04:00
Glenn Strauss 7c7f8c467c [multiple] split con, request (very large change)
NB: r->tmp_buf == srv->tmp_buf (pointer is copied for quicker access)

NB: request read and write chunkqueues currently point to connection
    chunkqueues; per-request and per-connection chunkqueues are
    not distinct from one another
      con->read_queue  == r->read_queue
      con->write_queue == r->write_queue

NB: in the future, a separate connection config may be needed for
    connection-level module hooks.  Similarly, might need to have
    per-request chunkqueues separate from per-connection chunkqueues.
    Should probably also have a request_reset() which is distinct from
2020-07-08 19:54:29 -04:00
Glenn Strauss 24680a9142 [core] array_init() arg for initial size 2020-07-08 18:08:52 -04:00
Glenn Strauss 5aadcba456 [core] buffer string in data_config
(instead of (buffer *))
2020-02-24 11:15:32 -05:00
Glenn Strauss a51278030e [core] const char *op in data_config
(instead of (buffer *))
2020-02-24 11:15:32 -05:00
Glenn Strauss c2238256e2 [core] inline array as part of data_array value
(instead of value being (array *))
2020-02-24 11:15:32 -05:00
Glenn Strauss 601c572c39 [core] inline buffer as part of data_string value
(instead of value being (buffer *))
2020-02-24 11:15:32 -05:00
Glenn Strauss ad9b7e009b [core] inline buffer as part of DATA_UNSET key
(instead of key being (buffer *))
2020-02-24 11:15:32 -05:00
Glenn Strauss 83535bbef3 [core] differentiate array_get_* for ro and rw
array_get_element_klen() is now intended for read-only access
array_get_data_unset() is used by config processing for r/w access
array_get_buf_ptr() is used for r/w access to ds->value (string buffer)
2020-02-24 11:15:32 -05:00
Glenn Strauss b2991c686d [core] perf: array.c performance enhancements
mark array_get_index() as hot, rewrite to be pure and return sorted pos

mark routines as pure, as appropriate

mark routines as cold if used only at startup for config processing

mark params const, as appropriate

array_get_buf_ptr() for modifiable value buffer after insert into array

uint32_t used and size members instead of size_t

remove a->unique_ndx member; simply add to end of array for value lists
remove du->is_index_key member; simply check buffer_is_empty(du->key)

array_insert_key_value() used to be a hint that lookup could be skipped,
but the state from array_get_index() is now saved and reused internally,
so the distinction is no longer needed.  Use array_set_key_value().
2020-02-24 11:15:32 -05:00
Glenn Strauss fb9b8ad8ae [core] mark startup/shutdown funcs cold 2019-02-04 02:25:48 -05:00
Glenn Strauss 449274903c [core] perf: simplify buffer_move()
require src and dest to be non-NULL

change no longer releases large swapped buffers with buffer_reset()
2018-11-25 19:51:07 -05:00
Glenn Strauss f69bd9cdb8 [core] perf: simple, quick buffer_clear()
quickly clear buffer instead of buffer_string_set_length(b, 0) or
buffer_reset(b).  Avoids free() of large buffers about to be reused,
or buffers that are module-scoped, persistent, and reused.

(buffer_reset() should still be used with buffers in connection *con
 when the data in the buffers is supplied by external, untrusted source)
2018-11-23 00:37:38 -05:00
Glenn Strauss 65fcd7810f [core] prefer buffer_append_string_len()
prefer buffer_append_string_len() when string len is known
(instead of buffer_append_string() which will recalculate strlen)
2018-09-23 19:18:49 -04:00
Glenn Strauss 8c7f1dfb03 [core] more memory-efficient fn table for data_*
save 40 bytes (64-bit), or 16 bytes (32-bit) per data_* element
at the cost of going through indirect function pointer to execute
methods.  At runtime, the reset() method is most used among them.
2018-09-23 18:01:58 -04:00
Glenn Strauss e8c1efd5df [core] data_config_pcre_compile,exec()
collect PCRE usage related to config processing
2018-08-05 03:44:15 -04:00
Glenn Strauss 04d76e7afd [core] some header cleanup
provide standard types in first.h instead of base.h
provide lighttpd types in base_decls.h instead of settings.h
reduce headers exposed by headers for core data structures
  do not expose <pcre.h> or <stdlib.h> in headers
move stat_cache_entry to stat_cache.h
reduce use of "server.h" and "base.h" in headers
2018-04-08 22:22:23 -04:00
Glenn Strauss 41eb4d323c [core] adjust parser for valid variable expansion
adjust config parser for valid variable expansion
Return only the value when a variable is expanded so that the
array element keeps its state as value-only or part of key-value

(thx nicorac)

2017-10-20 23:04:53 -04:00
Glenn Strauss b641850376 [core] optional condition in config "else" clause (fixes #1268)
  "condition should be optional in "else" clause in configuration file"
2017-05-15 22:02:33 -04:00
Glenn Strauss 86bb8be2c8 [core] perf: skip redundant strlen() if len known
performance: skip redundant strlen() if length is already known

introduce array_get_element_klen() to take key and klen params
2017-05-15 22:02:33 -04:00
Stefan Bühler 9752620792 [core] configparser: fix resource handling in error cases (fixes #2809)
- lemon never calls the destructor for variables on the RHS, make sure
  to manually clean up
- outside `if (ctx->ok) { }` always check for NULL pointers, i.e:
  - if (x) x->free(x)
  - buffer_free and array_free check for NULL on their own
- cleanup RHS variables below `if (ctx->ok) { }` at the bottom
  - set variables to NULL before if ownership gets passed on
- move some buffers instead of copying them

  "Memory corruption in yy_reduce (configparser.y), SIGSEGV"
2017-05-14 00:09:23 -04:00
Glenn Strauss 40f72a41b9 [core] omit default port from normalized host str
omit default scheme port from normalized host string
2017-04-25 11:12:53 -04:00
Glenn Strauss a53f662a30 [core] remove some unused header includes
remove exposure of stdio.h in buffer.h for print_backtrace(), now static
2017-03-28 02:17:33 -04:00
Philip Prindeville 367e62c1c2 [core] allow overriding prior config values (fixes #2799)
introduce ":=" config file syntax to replace previously set value

github: closes #78

  "allow overriding configuration values"
2017-03-23 01:29:05 -04:00
Glenn Strauss 2ac2911b9a [core] config match w/ arbitrary HTTP request hdrs (fixes #1556)
  "Allow matching against any arbitrary HTTP header in the configuration file regexps"
2017-01-31 14:36:16 -05:00
Glenn Strauss ce24523b59 [core] restrict where config "else" clauses occur (#1268)
(improve validation)

  "condition should be optional in "else" clause in configuration file"
2016-10-09 09:20:37 -04:00
Glenn Strauss 79fb75709b [core] optional condition in config "else" clause (fixes #1268)
  "condition should be optional in "else" clause in configuration file"
2016-10-09 08:06:41 -04:00
Glenn Strauss cd33554b74 [core] $HTTP["remoteip"] must handle IPv6 w/o []
[core] $HTTP["remoteip"] must handle IPv6 w/o [] (existing behavior)
This was inadvertently broken in lighttpd 1.4.40 when IP address
normalization was added.

In $HTTP["remoteip"], IPv6 is now accepted with or without '[]'.
http_request_host_normalize() expects IPv6 with '[]', and config
processing at runtime expects COMP_HTTP_REMOTE_IP compared without '[]',
so '[]' is stripped (internally) after normalization
2016-07-21 01:42:35 -04:00
Glenn Strauss df8032a7c8 use buffer_string_set_length() to truncate strings 2016-05-23 14:12:24 -04:00
Glenn Strauss e8dd8fdb53 [config] normalize IP strings in lighttpd.conf
normalize IP strings in lighttpd.conf conditionals:
  $SERVER["socket"], $HTTP["remoteip"], and $HTTP["host"]
2016-05-19 19:15:47 -04:00
Stefan Bühler dd33a350bb [configparser] fix small leak on config failure
From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-26 13:57:03 +00:00
Glenn Strauss f5453290b7 validate return values from strtol, strtoul (fixes #2564)
From: Glenn Strauss <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-26 10:58:49 +00:00
Stefan Bühler 5c68caa6d7 [core] replace array weakref with vector
From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-19 15:27:38 +00:00
Glenn Strauss 8abd06a7ff consistent inclusion of config.h at top of files (fixes #2073)
From: Glenn Strauss <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-19 15:14:35 +00:00
Glenn Strauss 43da581893 [core] configparser: error on duplicate keys in array merge (fixes #2685)
From: Glenn Strauss <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-15 18:56:05 +00:00
Stefan Bühler 68e4a416cc [core] provide array_extract_element and use it
From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-15 18:56:02 +00:00
Stefan Bühler e7a39cde36 [core] fix memory leak in configparser_merge_data
Release op1 memory on failure; fixes some theoretical memory leaks (a
failure results in early exit anyway).

From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-15 18:41:59 +00:00
Stefan Bühler 8d8ae9cbc8 [core] improve array API to prevent theoretical memory leaks
- refactor insert into array_find_or_insert; if the element already
  exists the caller must resolve the conflict manually:
  - array_replace frees the old element
  - array_insert_unique calls "insert_dup"
  both have no return value anymore
- fix usages of array_replace; they now don't need to delete the old
  entry anymore; usage in configparser was probably broken, as it
  possibly deleted the old element before calling array_replace

This should fix a lot of the issues reported in "Fortify Open Review
Project - lighttpd 1.4.39" (usually hitting the array limit):
when the array size limit was hit "new" entries leaked instead of
getting added.

On 32-bit INT_MAX entries cannot actually be reached (each entry
requires at least 48 bytes, leading to a total of 96GB memory).

On 64-bit INT_MAX entries would require 224GB memory, so it would be
theoretically possible. But it would need 2^27 reallocations of two
C-arrays of up to 16GB size.

From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-03-15 18:26:57 +00:00
Stefan Bühler 6c3c360129 [config] extend duplicate-array-key error (fixes #2704)
From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-02-23 17:05:10 +00:00
Stefan Bühler ad65603ec0 [core] fix conditional cache handling
- add new "skip" result to mark conditions that didn't actually get
  evaluated to false but just skipped because the preconditions failed.
- add "local_result" for each cache entry to remember whether the
  condition itself matched (not including the preconditions).
  this can be reused after a cache reset if the condition itself was not
  reset, but the preconditions were
- clear result of subtree (children and else-branches) when clearing a
  condition cache

From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-02-21 18:32:14 +00:00
Stefan Bühler 431559e5df [configparser] don't continue after parse error (fixes #2717)
only use values in reduce actions when the config is still valid

From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2016-02-19 15:49:06 +00:00
Stefan Bühler 6afad87d2e fix buffer, chunk and http_chunk API
* remove unused structs and functions
    (buffer_array, read_buffer)
  * change return type from int to void for many functions,
    as the return value (indicating error/success) was never checked,
    and the function would only fail on programming errors and not on
    invalid input; changed functions to use force_assert instead of
    returning an error.
  * all "len" parameters now are the real size of the memory to be read.
    the length of strings is given always without the terminating 0.
  * the "buffer" struct still counts the terminating 0 in ->used,
    provide buffer_string_length() to get the length of a string in a
    unset config "strings" have used == 0, which is used in some places
    to distinguish unset values from "" (empty string) values.
  * most buffer usages should now use it as string container.
  * optimise some buffer copying by "moving" data to other buffers
  * use (u)intmax_t for generic int-to-string functions
  * remove unused enum values: UNUSED_CHUNK, ENCODING_UNSET
  * converted BUFFER_APPEND_SLASH to inline function (no macro feature
  * refactor: create chunkqueue_steal: moving (partial) chunks into another
  * http_chunk: added separate function to terminate chunked body instead of
    magic handling in http_chunk_append_mem().
    http_chunk_append_* now handle empty chunks, and never terminate the
    chunked body.

From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2015-02-08 12:37:10 +00:00
Stefan Bühler 07dd0bd0a5 add force_assert() to enforce assertions as simple assert()s are disabled by -DNDEBUG (fixes #2546)
From: Stefan Bühler <>

git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2014-02-16 13:08:20 +00:00
Stefan Bühler 0bd06a9629 Fix segfault on invalid config "duplicate else conditions" (fixes #2065)
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2009-10-12 08:48:08 +00:00
Stefan Bühler 22e8b456a9 Fix header inclusion order, always include "config.h" before any system header
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2009-10-11 14:31:42 +00:00
Stefan Bühler 543f8040d3 Print an error if you use too many captures in a regex pattern (fixes #2059)
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2009-08-28 19:30:48 +00:00
Stefan Bühler 4df22f2a32 Fix issues found with clang analyzer
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2009-07-21 20:35:27 +00:00
Stefan Bühler 066b208bb6 Added Language conditional (fixes #1119); patch by petar
git-svn-id: svn:// 152afb58-edef-0310-8abb-c4023f1b3aa9
2009-02-05 22:36:58 +00:00