CompressionStream

The compression crate is Cruft's DEFLATE, gzip, zlib, and Brotli codec behind CompressionStream, DecompressionStream, and the node:zlib decode path. It compresses and decompresses real streams, and its decoders cap output and reject bad back-references so a small hostile stream cannot expand into gigabytes.

response.body.pipeThrough(new DecompressionStream("gzip")) sits on top of zlib in Node: decades old, fuzzed by every major distro, hardened against the class of input where a few kilobytes on the wire expand into gigabytes in memory. Cruft supplies the same surface with its own compression crate. This page is about what sits behind it, and where the hardening you inherit for free in Node has to be earned here instead.

Alpha, unaudited. The compression crate (Cruft 0.0.10) is a DEFLATE/gzip/zlib/brotli codec standing in for the CompressionStream/DecompressionStream web surface (and the node:zlib decode path) that Node backs with libdeflate and the brotli reference library. It has not been externally audited and has near-zero interop testing against real-world corpora, and it is not production-ready. Do not feed it adversarial input.

Codec plus stream wrapper

The web API is a stream, but a DecompressionStream is a buffering wrapper around a codec: it buffers, then hands whole buffers to a codec, then emits. So there are two surfaces to reason about, and they fail differently:

  • the codec (Huffman + LZ77 for DEFLATE, the RFC 7932 machinery for brotli): does it turn valid bytes into the right bytes, and does it refuse invalid bytes safely?
  • the stream wrapper: does it bound how much it holds in memory while it waits for the codec?

In Node both are libraries someone else hardened. Here they are two separate pieces, and the answer differs for each. The rest of this page keeps them apart.

What the codec does

The DEFLATE core is an independent RFC 1951 decoder: an LSB-first BitReader, a canonical HuffmanTable (from_lengths / decode), the fixed and dynamic block paths, and gzip (RFC 1952, gunzip), zlib (RFC 1950, zlib_inflate), and the HTTP Content-Encoding: deflate ambiguity handled by trying zlib then falling back to raw (http_deflate_inflate). CRC32 and Adler-32 checksums are verified on the gzip and zlib paths, and gunzip also checks the ISIZE trailer. This is real decode.

Encode exists too, and it is worth being precise about: the encoder is not just stored blocks. deflate_fixed and deflate_dynamic do genuine LZ77 (a hash-chain match finder) plus Huffman, with deflate_dynamic building per-block length-limited optimal trees via package-merge (package_merge, Larmore-Hirschberg). deflate_best picks the smallest of dynamic/fixed/stored. So output is standard DEFLATE any inflater accepts, and it actually compresses. (An older revision emitted stored blocks only; the current code does LZ77. The sibling rusty-js-deflate crate documented on the deflate page compresses the same way — a real LZ77 + Huffman encoder — so both this facade and that crate produce compressed output.)

Brotli is routed to the sibling rusty-js-brotli crate (brotli_decode, brotli_encode, the full Node options.params set via BrotliParams). The decode source flags its own boundary: it supports the exercised literal-only whole-buffer path plus known external literal test vectors; optimized dictionary/Huffman streams are intentionally not claimed. That is the crate marking where its brotli confidence ends.

What a test backs today:

  • brotli encode-then-decode round-trips at every quality 0..=11, empty input, all three modes, size-hint, and large-window (lgwin 30) (brotli_encode_quality_levels, brotli_encode_decode_roundtrip, brotli_params_*, brotli_encode_empty).
  • an external brotli test vector ("Hello, World!" from Python's brotli.compress) decodes correctly (brotli_decode_hello_roundtrip).
  • the DEFLATE size and back-reference guards fire (see the next section).

The bomb defenses and their tests

Every decoder has a size cap in the code: MAX_OUTPUT = 256 * 1024 * 1024 (256 MiB), checked before each output growth in decompress_block and on the stored-block path, plus DistanceTooFar rejection of a back-reference pointing before the start of output. Those are the two classic decompression-bomb defenses, and they are present.

A guard with no red test is inspection-strength only. Without a hostile test the bounds check rests on the author having read it correctly, which is exactly the trust libdeflate earned through fuzzing. Here the guards do have hostile tests:

  • inflate_with_limit_rejects_output_bomb_before_growth checks that inflate returns OutputTooLarge before the output actually grows past a caller limit.
  • inflate_rejects_bad_back_reference_before_output checks that a crafted bad back-reference returns DistanceTooFar before writing.
  • brotli got the same treatment: the brotli decoder used to cap output only per 64 KB meta-block with no total ceiling, a reachable bomb; now rusty-js-brotli exposes its own MAX_OUTPUT and decode_with_limit, compression exposes brotli_decode_with_limit, and brotli_decode_with_limit_preserves_output_cap checks that the accumulated-output cap fires.
  • The bomb and back-reference red tests exist at both the lower crate and this facade, and a fuzz lane is scaffolded (compression_decoders, deflate_stream, tar_archive targets).

The stream wrapper is the third piece: the DecompressionStream adapter buffers all compressed input whole before decoding, a second input-side memory amplifier. That is bounded at the host node:zlib surface: a 16 MiB compressed-input cap on the whole-buffer adapter, a 64 MiB default decode ceiling (lower than the codec's 256 MiB), and a 128x expansion-ratio cap on stream finish. Note where those numbers live: the tighter, per-request-sane defaults are enforced at the node:zlib/host wiring, not baked into this crate's 256 MiB MAX_OUTPUT constant. A direct call to compression::inflate gets the 256 MiB cap, not the 64 MiB one.

What it does not do

  • It is not an interop-proven codec. The tests are round-trips and a handful of known-answer tests. There is no broad corpus of real-world gzip/brotli streams from live servers, no differential against zlib/libdeflate across thousands of inputs. Round-tripping its own encoder proves the encoder and decoder agree with each other, not that either agrees with the rest of the internet.
  • Brotli decode is deliberately partial. Optimized dictionary and Huffman brotli streams are intentionally not claimed. A real Content-Encoding: br response from a tuned production server is exactly the case this path does not promise to handle. This is stated in-source.
  • No preset-dictionary support on the zlib path (FDICT is rejected), and gzip only accepts compression method 8.
  • The direct-call cap is coarse. compression::inflate / gunzip / zlib_inflate cap at 256 MiB and are per-call: concurrent hostile requests each allocate up to that. The sane server defaults live one layer up at the host surface, so a direct consumer of this crate does not inherit them.

How it compares to what it replaces

zlib/libdeflate and the brotli reference library are the maturity baseline: decades of fuzzing, formal-ish interop, every edge of the RFCs exercised by the whole ecosystem. This crate matches their surface and their format (output is standard DEFLATE/gzip/zlib/brotli that any conforming decoder reads), and it matches their bomb-resistance posture on the paths that have tests. It does not match their interop confidence: correctly decoding whatever a real server sent, the thing you got for free from zlib, is precisely the unproven part here.

Limitations

The codec is real (genuine LZ77 + package-merge Huffman, not a stored-block stub), and the security story is solid where tested: the brotli bomb, the deflate caps, and the stream input amplifier all have red tests, and the tighter per-request ceilings and ratio cap live at the host surface. What remains unproven is not the presence of defenses but the breadth of correctness: interop against real-world streams is near-zero, brotli decode is explicitly partial, and the caller-facing compression::* entry points still carry the coarse 256 MiB per-call cap rather than the server-sane defaults. Trust it to refuse a bomb on a tested path. Do not yet trust it to decode an arbitrary production br response, and do not assume a direct crate call inherits the host's tighter limits.