lz4

LZ4 compression — ultra-fast algorithm, block and streaming compression/decompression.

Module lz4 | Source packages/front/fw/src/io/compress/lz4.js | Deps none | Worker-safe yes

Resolve

const lz4 = runtime.resolve('lz4');
// Returns: { compress, decompress, Lz4CompressStream, Lz4DecompressStream }

API

lz4.compress(src, len, opts?) → [size, data]

Compresses a Uint8Array block.

const [size, data] = lz4.compress(uint8Array, uint8Array.length);                      // mode 'speed'
const [rSize, rData] = lz4.compress(uint8Array, uint8Array.length, { mode: 'ratio' }); // smaller, slower

Throws a RangeError naming the value when opts.mode is neither 'speed' nor 'ratio'. Both modes produce standard LZ4 blocks: decompress reads either.

lz4.decompress(src) → [size, data]

Decompresses an LZ4 block.

const [origSize, original] = lz4.decompress(compressed);

new lz4.Lz4CompressStream(ondata?, opts?)

Streaming compression. Each chunk is processed as an independent LZ4 block. opts.mode is the same as for compress (default 'speed'); an unknown mode throws a RangeError at construction.

const chunks = [];
const stream = new lz4.Lz4CompressStream((chunk) => chunks.push(chunk));

stream.push(data1);
stream.push(data2);
stream.push(lastChunk, true);  // true = dernier chunk

new lz4.Lz4DecompressStream(ondata?)

Streaming decompression.

const output = [];
const stream = new lz4.Lz4DecompressStream((chunk) => output.push(chunk));

stream.push(compressedChunk);
stream.push(lastCompressedChunk, true);

Characteristics

Modes

Both modes run the same encoder and emit the same block format; they differ only in how hard the match search works.

Measured with the codec bench (tools/bench/codec-levels.js --lz4-mode, Bun 1.3.13, 256 KiB corpora, small = 200 × 2 KiB slices; encode time in ms, median of 7), next to the single-mode encoder this module replaced (pre-rewrite baseline):

'speed' is never larger than the previous encoder on these corpora and encodes at about its speed (from −85 % on many small inputs to +20 % on short-match text such as json); 'ratio' trades 1.5 to 1.8 times the encode time on large text for blocks 8 to 17 % smaller. Both decode with the same decompress, within about 10 % of each other (a 'speed' block carries more literals, a 'ratio' block more matches).

Full example

const lz4 = runtime.resolve('lz4');
const utf8 = runtime.resolve('utf8');

const data = utf8.toBytes('Hello World '.repeat(1000));

// Block
const [size, compressed] = lz4.compress(data, data.length);
console.log(`${data.length} → ${size} bytes`);

const [origSize, restored] = lz4.decompress(compressed);
console.log(utf8.fromBytes(restored));  // 'Hello World Hello World ...'

// Streaming compression
const parts = [];
const comp = new lz4.Lz4CompressStream((chunk) => parts.push(chunk));
comp.push(data.slice(0, 500));
comp.push(data.slice(500), true);

Worker Usage

const worker = fw.createWorker(
    function ({ libs, args }) {
        const lz4 = libs.lz4;
        const [size, compressed] = lz4.compress(args.data, args.data.length);
        self.postMessage({ size, compressed }, [compressed.buffer]);
    },
    { dependencies: ['lz4'], args: { data: new Uint8Array(rawData) } }
);

Notes

• LZ4 favours speed (hundreds of MB/s) over ratio — for maximum ratio, use brotli (quality 11) or deflate (level 9).
• Block format: each compress call produces an independent LZ4 block — no multi-block LZ4 frame header.
• Lz4CompressStream / Lz4DecompressStream: each push produces one or more complete blocks (no inter-push buffering).

Design

Match search (hash, chain, skip). Every visited position hashes its 4-byte little-endian word multiplicatively (Math.imul by 2654435761, top hashBits bits, with hashBits between 8 and 16 sized to the input) into a bucket table. In 'ratio' mode each bucket heads a chain linked through a 64 KiB ring indexed by position, so a lookup walks older candidates newest first: at most 4 inside the 64 KiB window, keeping the longest and stopping early on a match of 32 bytes or one that reaches the end limit. In 'speed' mode only the newest entry of the bucket is tried, and no chain is kept at all (no ring is allocated or written). A cheap test comes first: the byte at the current best length must match before the whole 4-byte word is compared. The kept match is then widened backwards over the pending literals (both modes: without it, 'speed' would exceed the previous encoder's size on the text corpora). After a match, 'ratio' indexes its last 3 covered positions, 'speed' none. After a run of misses the stride grows by one every 64 ('ratio') or 32 ('speed') misses, so incompressible stretches are sampled instead of scanned. The two modes are one function with three parameters (probes, covered positions, skip shift); 'ratio' is byte-identical to the single-mode encoder it replaced, and the parameterisation costs it nothing measurable.

Match extension. Past the first 4 bytes, a candidate is compared byte by byte up to 16 bytes, because most matches end there. A candidate that gets that far is compared in 4-byte words, and a final byte loop finishes whatever the word loop leaves before the end limit. The length found is always the exact common prefix; lz4.test.js checks it for every run length from 7 to 96 and at the end-of-block cut.

Literal copies. Literals are copied byte by byte in both the encoder and the decoder. Literal runs between matches are short, and a set(subarray(...)) would allocate a view object for every sequence (a Buffer when the input is one), which costs more than the copy it saves.

What the chain costs. The 4-probe chain and the 3-position indexing are what buy 'ratio''s smaller blocks; 'speed' drops both and keeps only a fraction of the size gain (see Modes). Among the configurations measured for 'speed' (1 or 2 probes, 0 to 3 indexed positions, with and without backward widening, skip after 16, 32 or 64 misses, 14 to 16 hash bits), the one shipped is the fastest overall that stays at or below the previous encoder's size on every bench corpus.

Decoder. Two passes: the first validates every sequence and sums the output size (so the size limit is checked before any allocation), the second copies into one exact-size buffer. A non-overlapping match longer than 32 bytes uses copyWithin. Overlapping matches are copied byte by byte, because they re-read bytes they have just written.

Benchmark

tools/bench/codec-levels.js --lz4-mode {speed,ratio}, --reps 7 --warmup 3, baseline sha d47f8fc7a68809f519fe7d36ff2b0ba88bcacdbd (the single-mode encoder this module replaced). Host win32 / 13th Gen Intel(R) Core(TM) i7-13700KF / Bun 1.3.13. Generated 2026-09-24T21:14:48.449Z (speed) and 2026-09-24T21:14:48.950Z (ratio), both gitignored bench artifacts.

Gate reading (a later measurement ruling). The rule for lz4 is Δbytes ≤ 0 per corpus — the only gated figure; the mode's own encode time is recorded, not gated: 'speed''s delta is the accepted measurement floor (the fastest configuration found that still keeps bytes ≤ 0 on every corpus), and 'ratio''s delta is a recorded compression trade-off. Both decode with the unchanged decompress.

'speed' (default)

Bytes ≤ 0 on all 6 corpora. The reps-7/reps-21 spread on text/source/json (companion artifact 05-supp-lz4-speed-reps21-20260924T2114Z.json, gitignored) is the harness's own noise floor — the baseline side alone moved from 0.82 to 0.65 ms on text between the two runs — not a regression; it is accepted as the measured floor by the ruling.

'ratio'

Bytes ≤ 0 on all 6 corpora. Encode time (+44 to +83 % on text/source/json) is recorded as the ratio trade-off, not gated — this is the same encoder the module shipped before the speed/ratio split, byte-identical by hash (gitignored companion artifact, reps 21).

Totals: 12/12 PASS on the gated figure (bytes ≤ 0, both modes, all 6 corpora); 0 cells fail on bytes. No byte was traded for time in the default: 'speed' stays at or below the previous encoder's size on every corpus (see Modes for the configuration search).

Provenance

In-house implementation written from the LZ4 Block Format specification; the earlier node-lz4-derived implementation was replaced by a clean-room rewrite (2026-09).

See also

• gzip, zlib — standard HTTP compression
• deflate — DEFLATE engine