Commit Graph

86 Commits

Author SHA1 Message Date
Glenn Strauss 9a2404cec6 [core] quiet compiler warnings
cast away signedness warning in request_check_hostname()
mod_gnutls https_add_ssl_client_entries crts allocated if crt_size != 0
  (which is already checked earlier in routine)

report from FaceBook Infer static analysis tool (
- quiet dead store warnings
- check return != NULL from allocation funcs
2021-09-04 08:08:26 -04:00
Glenn Strauss f4ff56e064 [mod_deflate] use http_header_str_contains_token 2021-09-04 08:08:26 -04:00
Glenn Strauss 7a913fb153 [mod_deflate] improve compress.*->deflate.* remap 2021-08-27 02:16:54 -04:00
Glenn Strauss a3603074c1 [mod_deflate] use brotli quality 5 by default
BROTLI_DEFAULT_QUALITY is 11, which may lead to a higher compression
ratio, but potentially at a cost of taking *many* multiples of the
time taken to compress at quality level 5.

2021-08-27 02:16:54 -04:00
Glenn Strauss 3f248f0900 [mod_deflate] deflate.params per-encoder params
mechanism to define per-encoder params
parsed into structured data at startup

compression level is the initial target
deflate.params is a better solution to the deflate.compression-level,
which is a single range 1-9 that is overload and applied to all encoders
without any scaling, even though encoders might have different scales.

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 777d659c5c [mod_deflate] use ZSTD_c_strategy w/ compress lvl
map deflate.compression-level to ZSTD_c_strategy as the current
enum for ZSTD_strategy is in the range 1 .. 9, which matches the
gzip compression level 1 .. 9 range

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 7759660de5 [mod_deflate] use deflate.allowed-encodings order
use deflate.allowed-encodings ordering as server preference list
when choosing an encoding from list in client request Accept header

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 1cd73b08a6 [core] move backtrace and assert macros to ck.[ch] 2021-08-27 02:16:53 -04:00
Glenn Strauss af3df29ae8 [multiple] reduce redundant NULL buffer checks
This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.

Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.

In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func.  In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.

- check for empty strings at config time and set value to NULL if blank
  string will be ignored at runtime; at runtime, simple pointer check
  for NULL can be used to check for a value that has been set and is not
  blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
  and use buffer_is_unset() instead of buffer_is_empty(),
  where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
  known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
  truncate string, and use buffer_extend() to extend

Examples where buffer known not to be NULL:
  - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
    (though we might set it to NULL if buffer_is_blank(cpv->v.b))
  - address of buffer is arg (&foo)
    (compiler optimizer detects this in most, but not all, cases)
  - buffer is checked for NULL earlier in func
  - buffer is accessed in same scope without a NULL check (e.g. b->ptr)

internal behavior change:
  callers must not pass a NULL buffer to some funcs.
  - buffer_init_buffer() requires non-null args
  - buffer_copy_buffer() requires non-null args
  - buffer_append_string_buffer() requires non-null args
  - buffer_string_space() requires non-null arg
2021-08-27 02:16:53 -04:00
Glenn Strauss 13ea2d880b [core] consistent inclusion of sys-time.h 2021-04-28 14:08:29 -04:00
Glenn Strauss 680e6b3bca [multiple] buffer_copy_path_len2() aggregate 2021-04-02 01:16:42 -04:00
Glenn Strauss dc01487ea6 [multiple] use buffer_append_* aggregates
reduces the number of round-trips into some frequently-called routines
2021-04-02 01:16:40 -04:00
Glenn Strauss 26f354cb37 [multiple] http_header APIs to reduce str copies 2021-03-26 22:38:36 -04:00
Glenn Strauss 7df6fb5ec8 [mod_deflate] do not cache 206 Partial Content 2021-02-24 20:46:44 -05:00
Glenn Strauss 5b0aed8c32 [mod_deflate] compat with zstd < v1.4.0
ZSTD_compressStream2() was an "advanced API" (experimental; unstable)
in v1.3.x
2021-01-17 08:33:02 -05:00
Glenn Strauss f8792bfb5a [mod_deflate] use zstd typedefs (minor cleanup) 2021-01-17 08:32:38 -05:00
Glenn Strauss 8ae6807654 [mod_deflate] use zstd streaming API 2021-01-12 22:39:28 -05:00
Glenn Strauss 9211fb3d86 [mod_deflate] support Accept-Encoding: zstd 2021-01-12 18:29:20 -05:00
Glenn Strauss b700a8ca09 [multiple] etag.[ch] -> http_etag.[ch]; better imp
more efficient implementation of HTTP ETag generation and comparison

modify dekhash() to take hash value to allow for incremental hashing
2020-12-25 14:41:16 -05:00
Glenn Strauss af04e0b0e1 [core] silence coverity warnings (false positives) 2020-10-20 15:29:25 -04:00
Glenn Strauss 019c513819 [multiple] use http_chunk_append_file_ref()
use http_chunk_append_file_ref() and http_chunk_append_file_ref_range()

reduce resource usage (number of fds open) by reference counting open
fds to files served, and sharing the fd among FILE_CHUNKs in responses
2020-10-20 11:51:48 -04:00
Glenn Strauss e057c5413b [mod_deflate] use large mmap chunks to compress
use large chunks since server blocks while compressing, anyway
(mod_deflate is not recommended for large files)
2020-10-20 11:51:47 -04:00
Glenn Strauss 3f1a12e5fb [multiple] remove chunk file.start member
c->offset is now offset into file
c->file.length is end of octets to send (end pos + 1)
c->file.length - c->offset is num of octets to send
2020-10-19 21:40:14 -04:00
Glenn Strauss 66d1ec485c [core,mod_deflate] leverage cache of open fd
leverage simple cache of open file in stat_cache
(use stat_cache_get_entry_open())

future: reference count fd instead of dup()
  (requires extending chunkqueue interfaces)
2020-10-19 21:40:14 -04:00
Glenn Strauss e2c832122b [mod_deflate] fix potential NULL deref in err case
(bug on master branch; never released)
2020-10-11 12:19:27 -04:00
Glenn Strauss 4d6d1e790a [multiple] use light_btst() for hdr existence chk 2020-10-11 12:19:27 -04:00
Glenn Strauss ee1f7af24d [core] be more precise checking streaming flags 2020-10-11 12:19:27 -04:00
Glenn Strauss 81029b8b51 [multiple] inline chunkqueue where always alloc'd
inline struct chunkqueue where always allocated in other structs

(memory locality)
2020-10-11 12:19:27 -04:00
Glenn Strauss 9c8981a7d1 [core] tst,set,clr macros for r->{rqst,resp}_htags 2020-10-11 12:19:26 -04:00
Glenn Strauss 5fd8a26a75 [core] defer optimization to read small files
defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured
2020-10-11 12:19:24 -04:00
Glenn Strauss 4b755d2b03 [mod_deflate] quicker check for Content-Encoding
The response is not a candidate for mod_deflate if
Content-Encoding was already set by something else.
2020-08-02 07:47:42 -04:00
Glenn Strauss 33c8cf41db [multiple] rename connection_reset hook to request
rename connection_reset to handle_request_reset
2020-08-02 07:47:41 -04:00
Glenn Strauss 8dd33a72dd [mod_deflate] mod_deflate subsumes mod_compress
translate config server.modules "mod_compress" to "mod_deflate"
accept compress.* directives, but issue DEPRECATED warning trace

mod_deflate differences from mod_compress:
- mod_compress compress.filetype was exact match; deflate.mimetypes is
  prefix match (behavior change might compress longer mimetype matches,
  which are likely of similar type and compressability)
- mod_compress always sent entire (compressed) file for Range request
  mod_deflate will stream compress range result (not stored in cache)
- mod_compress would short-circuit request with 403 Forbidden error
  if request file did not exist (stat() failed) (This behavior was
  unfriendly to other handlers)
- mod_compress compress.cache-dir layout differs from deflate.cache-dir
  layout; file cache should be cleared (or renamed) when migrating from
  mod_compress to mod_deflate
- mod_deflate does not issue Vary: Accept-Encoding if request does not
  contain Accept-Encoding.  The identity response can be cache by
  proxies and served to clients.  Historically, some proxies disabled
  caching if any Vary: response was seen.  If the Vary header is
  desirable, mod_deflate code which checks for Accept-Encoding and
  compression type can be moved down a few lines to be below the
  setting of the Vary response header.
2020-08-02 06:46:22 -04:00
Glenn Strauss be6da785c5 [mod_deflate] deflate.cache-dir compressed cache
(achieves feature parity with mod_compress)
2020-08-02 06:46:22 -04:00
Glenn Strauss 5b15c1f449 [mod_deflate] propagate errs from internal funcs
also, pass one less arg to internal funcs
2020-08-02 06:46:22 -04:00
Glenn Strauss b2fe37f0a9 [mod_deflate] fix typo in config option
(typo on master branch; never released)
2020-08-02 06:46:22 -04:00
Glenn Strauss 8d5e237c60 [mod_deflate] Brotli support
configuration option: ./configure --with-brotli
2020-08-02 06:46:22 -04:00
Glenn Strauss c752d4696e [multiple] correct misspellings in comments
  "Script for fixing spelling errors with codespell"
2020-07-08 19:54:30 -04:00
Glenn Strauss a0029b21a1 [core] remove r->uri.path_raw; generate as needed
(r->uri.path_raw previously duplicated from r->target, minus query-part)
2020-07-08 19:54:29 -04:00
Glenn Strauss 7c7f8c467c [multiple] split con, request (very large change)
NB: r->tmp_buf == srv->tmp_buf (pointer is copied for quicker access)

NB: request read and write chunkqueues currently point to connection
    chunkqueues; per-request and per-connection chunkqueues are
    not distinct from one another
      con->read_queue  == r->read_queue
      con->write_queue == r->write_queue

NB: in the future, a separate connection config may be needed for
    connection-level module hooks.  Similarly, might need to have
    per-request chunkqueues separate from per-connection chunkqueues.
    Should probably also have a request_reset() which is distinct from
2020-07-08 19:54:29 -04:00
Glenn Strauss 31d9495330 [core] store subrequest_handler instead of mode
store pointer to module in handler_module instead of con->mode id
2020-07-08 19:54:29 -04:00
Glenn Strauss 100dfaa3f3 [core] move plugin_ctx into (request_st *)
NB: in the future, a separate plugin_ctx may be needed for
    connection-level plugins to keep state across multiple requests
2020-07-08 19:54:29 -04:00
Glenn Strauss 1474be7859 [core] move addtl request-specific struct members 2020-07-08 19:54:29 -04:00
Glenn Strauss 19985261b2 [core] convenience macros to check req methods 2020-07-08 19:54:29 -04:00
Glenn Strauss 03b4c993d2 [multiple] generic config array type checking 2020-07-08 19:54:28 -04:00
Glenn Strauss f24e6d696a [multiple] plugin_stats array
use global rather than passing around (server *) just for that

li_itostrn() and li_utostrn() return string length
(rather than requiring subsequent strlen() to find length)
2020-07-08 19:54:28 -04:00
Glenn Strauss 50bdb55de8 [multiple] connection hooks no longer get (srv *)
(explicit (server *) not passed; available in con->srv)
2020-07-08 19:54:28 -04:00
Glenn Strauss 010c28949c [multiple] prefer (connection *) to (srv *)
convert all log_error_write() to log_error() and pass (log_error_st *)

use con->errh in preference to srv->errh (even though currently same)

avoid passing (server *) when previously used only for logging (errh)
2020-07-08 19:54:28 -04:00
Glenn Strauss b73949e03f [multiple] plugin.c handles common FREE_FUNC code
(simpler for modules; less boilerplate to cut-n-paste)
2020-07-08 18:08:51 -04:00
Glenn Strauss ed62e354ff [core] use config_plugin_values_init() 2020-07-08 18:08:51 -04:00