Zstandard Compression: Data Compression Deep Dive

1. Introduction and Core Concepts

Zstandard compression, commonly abbreviated as Zstd or ZSTD, is a modern lossless data compression algorithm developed by Yann Collet at Facebook (now Meta) and first released in 2016. Designed to provide a superior trade-off between compression ratio, compression speed, and decompression speed, zstandard compression has rapidly become a cornerstone of modern data infrastructure. It is an open-source, RFC 8478 standardized algorithm that combines proven entropy coding techniques with innovative optimizations to deliver performance that adapts gracefully across a wide spectrum of use cases, from real-time network communication to long-term archival storage.

At its core, zstandard compression is a dictionary-based, LZ77-style compressor that belongs to the same family as DEFLATE (used in gzip and zlib) and LZ4, but it significantly advances the state of the art. Unlike older algorithms that force a harsh choice between speed and ratio, Zstd offers 22 default compression levels (1 to 22) plus ultra levels up to 22 and negative levels for even faster compression. This tunability allows developers to precisely dial in the desired balance: level 1 rivals LZ4 in speed while still offering better compression, while level 19-22 competes with xz and brotli for maximum density. Decompression speed, crucially, remains remarkably fast and nearly independent of the compression level chosen.

The importance of zstandard compression stems from the exponential growth of data. In an era where web assets, database logs, container images, and backups must be moved and stored efficiently, the cost of compression is measured not just in bytes saved but in CPU cycles and latency. Zstd addresses this by delivering high decompression throughput (often exceeding 1 GB/s per core), low memory footprint, and a unique feature: dictionary compression for small data. This ability to train a shared dictionary on small, similar payloads like JSON records or API responses allows Zstd to achieve compression ratios on tiny messages that traditional algorithms cannot match, making it ideal for modern microservices and storage systems.

Core Design Goals

Zstandard was built around four explicit goals: high compression ratio, very fast decompression, tunable compression speed, and scalability. The algorithm achieves this by decoupling the search for redundancy from the encoding of that redundancy. It uses a large, configurable window size (up to 128 MB, and even larger with long-distance matching) to find duplicate strings far apart in the data, and then encodes the resulting literals and matches with a highly optimized entropy coder called Finite State Entropy (FSE) combined with Huffman coding. This hybrid approach is more efficient than the Huffman-only coding of DEFLATE.

Another foundational concept is frame-based organization. Every Zstd compressed payload is encapsulated in one or more frames, each containing a header, optional dictionary ID, blocks, and a checksum. Frames can be concatenated, support streaming, and allow for skippable frames for metadata. This structure makes the format robust for streaming compression, file archiving, and network protocols where data may be processed incrementally without knowing the total size in advance.

2. How It Works

Zstandard compression operates as a multi-stage pipeline, where each stage transforms the data to expose and then eliminate redundancy. Understanding this pipeline is key to appreciating why Zstd outperforms its predecessors. The process can be broken down into five distinct phases: parsing and matching, sequence encoding, block partitioning, entropy coding, and frame formatting.

The first and most computationally intensive stage is LZ77 parsing. The compressor scans the input through a sliding window and searches for repeated sequences. When it finds a previous occurrence of the current byte sequence, it replaces that sequence with a match represented as an (offset, length) pair. For example, the string "abcabcabc" might be encoded as literals "abc" followed by a match with offset 3 and length 6. Zstd uses highly optimized match finders, including hash tables, binary trees, and hash chains, selected dynamically based on the compression level. Higher levels perform more exhaustive, near-optimal parsing (including lazy and optimal parsing strategies) to find the best combination of matches, while lower levels use fast hashing for speed.

Once the input is converted into a sequence of literals and matches, Zstd does not immediately entropy code them. Instead, it encodes these sequences themselves. Literals are the raw bytes that could not be matched, while sequences consist of three components: literal length, match offset, and match length. Zstd can encode offsets using either a dedicated offset code table or as a direct offset, and it reuses previous offsets via a repcode mechanism similar to LZ4, which significantly improves compression of structured data. This separation allows each component to be compressed with the most suitable model.

Finite State Entropy and Huffman Coding

The heart of Zstd's entropy coding is Finite State Entropy (FSE), a variant of Asymmetric Numeral Systems (ANS) developed by Jarek Duda. FSE is faster than traditional Huffman coding and achieves compression ratios close to arithmetic coding. Zstd uses FSE to compress the sequence codes (literal lengths, match lengths, and offsets) and uses Huffman coding for literals. In many cases, it builds custom FSE and Huffman tables per block, adapting to the local statistics of the data. For blocks where building a table would be overhead, it can reuse previous tables or use a predefined default.

Before entropy coding, Zstd performs block partitioning. The input is split into blocks (typically up to 128 KB each), and each block can be compressed independently with one of four block types: Raw (uncompressed), RLE (single repeated byte), Compressed (using the full pipeline), or Reserved. The compressor analyzes each block and chooses the most efficient representation. This adaptivity ensures that incompressible data does not expand significantly and that highly redundant blocks are handled optimally.

The Role of Dictionaries and Long Distance Matching

Two advanced features distinguish Zstd: dictionary compression and Long Distance Matching (LDM). Dictionary compression pre-trains FSE/Huffman tables and a shared content dictionary on a sample of similar small files. When compressing small payloads (e.g., 1KB JSON documents), the dictionary provides a pre-seeded window of common substrings, dramatically improving ratio. Long Distance Matching extends the maximum match distance from the default window (8 MB at high levels) to up to 128 MB or more, enabling Zstd to find redundancy across very large files, such as VM images or genomic data, by using a separate hash table for long-range matches.

3. Implementation

While the reference implementation is in C, Python provides excellent bindings through the zstandard library (by Gregory Szorc) and the simpler zstd module. These bindings expose the full power of the algorithm, including streaming, dictionaries, and advanced parameters. The following examples demonstrate idiomatic, production-ready usage.

The most common use case is simple one-shot compression and decompression. This is suitable for compressing files, database blobs, or API payloads that fit in memory. Note how the compressor object is reusable and level selection directly impacts ratio and time.

import zstandard as zstd

# Sample data: repetitive log-like text benefits greatly from zstd
data = (b"INFO: User 12345 connected from 192.168.1.1\n" * 1000 +
        b"ERROR: Timeout on request id=abc-xyz\n" * 200)

# --- One-shot compression ---
# Level 3 is the default, balanced choice. Range is 1 (fastest) to 22 (best ratio)
cctx = zstd.ZstdCompressor(level=3)
compressed = cctx.compress(data)

print(f"Original: {len(data)} bytes")
print(f"Compressed: {len(compressed)} bytes")
print(f"Ratio: {len(compressed) / len(data):.3f}")

# --- One-shot decompression ---
dctx = zstd.ZstdDecompressor()
decompressed = dctx.decompress(compressed)

assert decompressed == data
# For untrusted data, always set max_output_size to prevent decompression bombs
# decompressed = dctx.decompress(compressed, max_output_size=50 * 1024 * 1024)

For large files or network streams, loading everything into memory is inefficient. Zstd's frame format is inherently streamable, allowing incremental compression and decompression with constant memory. This is essential for web servers, log pipelines, and backup tools.

import zstandard as zstd
import io

def compress_stream(input_path, output_path, level=5):
    """Compress a file using streaming API to keep memory usage low."""
    cctx = zstd.ZstdCompressor(level=level, write_checksum=True)
    
    with open(input_path, 'rb') as fin, open(output_path, 'wb') as fout:
        # stream_reader yields compressed chunks as we read
        with cctx.stream_reader(fin) as reader:
            while True:
                chunk = reader.read(16384)  # 16 KB chunks
                if not chunk:
                    break
                fout.write(chunk)

def decompress_stream(input_path, output_path):
    """Decompress a stream, verifying checksum if present."""
    dctx = zstd.ZstdDecompressor()
    with open(input_path, 'rb') as fin, open(output_path, 'wb') as fout:
        # stream_reader handles concatenated frames automatically
        with dctx.stream_reader(fin) as reader:
            while True:
                chunk = reader.read(16384)
                if not chunk:
                    break
                fout.write(chunk)

# Example: compress in memory using stream_writer for network sockets
def compress_for_socket(data_iter):
    cctx = zstd.ZstdCompressor(level=1)  # Level 1 for real-time speed
    compressor = cctx.stream_writer(io.BytesIO())
    # In a real socket, you would write to socket.makefile()
    for piece in data_iter:
        # < and > must be escaped in HTML code blocks
        if len(piece) > 0 and len(piece) < 1024 * 1024:
            compressor.write(piece)
    return compressor.flush()

Dictionary compression is where Zstd truly shines for small records. Training a dictionary requires a representative sample set (typically 100+ samples, total size ~100x dictionary size). Once trained, the same dictionary must be shared between compressor and decompressor.

import zstandard as zstd

# Simulate 1000 small JSON records with similar structure
samples = [b'{"user_id": %d, "action": "click", "page": "/home", "ts": 1710000000}' % i for i in range(1000)]

# Train a 8KB dictionary from samples
# dict_size should be ~ 1-5% of total sample size for best results
dictionary = zstd.train_dictionary(8192, samples)

print(f"Dictionary size: {len(dictionary.as_bytes())} bytes")
print(f"Dictionary id: {dictionary.dict_id()}")

# Compress a new, similar record with and without dictionary
new_record = b'{"user_id": 9999, "action": "click", "page": "/home", "ts": 1710000999}'

cctx_no_dict = zstd.ZstdCompressor(level=3)
cctx_with_dict = zstd.ZstdCompressor(level=3, dict_data=dictionary)

compressed_no_dict = cctx_no_dict.compress(new_record)
compressed_with_dict = cctx_with_dict.compress(new_record)

print(f"Without dict: {len(compressed_no_dict)} bytes")
print(f"With dict: {len(compressed_with_dict)} bytes")

# Decompression MUST use the same dictionary
dctx = zstd.ZstdDecompressor(dict_data=dictionary)
assert dctx.decompress(compressed_with_dict) == new_record

For advanced tuning, Zstd exposes compression parameters that allow fine-grained control over window size, hash log, chain log, and search depth. This is useful when optimizing for specific data types or memory constraints.

import zstandard as zstd
import time

data = b"abcdefghijklmnopqrstuvwxyz" * 50000  # Highly compressible

# Benchmark different levels
for level in [1, 3, 6, 9, 15, 19]:
    cctx = zstd.ZstdCompressor(level=level)
    start = time.perf_counter()
    compressed = cctx.compress(data)
    elapsed = time.perf_counter() - start
    ratio = len(compressed) / len(data)
    print(f"Level {level:2d}: {len(compressed):6d} bytes ratio={ratio:.3f} time={elapsed*1000:.2f}ms")

# Custom advanced parameters: small window for low-memory decompression
# Useful for embedded systems where decompressor memory is limited
params = zstd.ZstdCompressionParameters(
    window_log=18,  # 256 KB window instead of default 23 (8 MB)
    chain_log=14,
    hash_log=15,
    search_log=4,
    min_match=4,
    target_length=32,
    strategy=zstd.STRATEGY_BTOPT  # Binary Tree Optimal parsing
)
cctx_custom = zstd.ZstdCompressor(compression_params=params)
compressed_custom = cctx_custom.compress(data)
print(f"Custom params size: {len(compressed_custom)}")

# Check decompressor memory requirement
# if window_log < 20, decompression needs < 1 MB

4. Step-by-Step Example

To make the pipeline concrete, let us walk through the compression of a short ASCII string: TOBEORNOTTOBEORTOBEORNOT (24 bytes). This phrase contains clear repetition ("TOBEOR" and "NOT") that Zstd will exploit. While a real block would be larger, this example illustrates each transformation.

Step 1: Input and Block Splitting. The input is treated as a single block since it is far smaller than the default block size. The compressor first decides the block type. Since the data is not incompressible and not a single repeated byte, it will be a Compressed block.

Step 2: LZ77 Parsing and Sequence Generation. The match finder scans left to right. It maintains a hash table of recent positions. At position 0, it emits literals "T", "O", "B", "E", "O", "R", "N", "O", "T" (9 literals) until it finds a match at position 9. At position 9, the substring "TOBEOR" matches the substring at offset 9 (distance back to position 0) with length 6. The parser emits a sequence: literal_length=9, offset=9, match_length=6. The next bytes are "TOBEOR" again? Actually remaining is "TOBEORNOT" - at position 15, "TOBEOR" matches again at offset 6 (or 15) with length 6, then "NOT" matches at offset 15 with length 3. The final sequence set is: literals="TOBEORNOT", seq1=(offset=9, match_len=6), seq2=(offset=9, match_len=9) depending on optimal parsing. The exact parsing depends on level, but the core idea is conversion to literals + sequences.

Step 3: Sequence Encoding. The sequences are encoded. Literal lengths (9), offsets (9), and match lengths (6, 9) are each assigned codes from their respective FSE tables. Offsets are often encoded with an extra bits scheme: an offset code plus additional bits to represent the precise value. Repcodes are checked: if the offset repeats, a small repcode (0-2) is used instead of a full offset, saving bits.

Step 4: Entropy Coding. The literals "TOBEORNOT" are Huffman encoded. Since letters like 'O' and 'T' appear 6 and 4 times respectively, they receive shorter Huffman codes (e.g., 2-3 bits) than rarer letters. The sequence codes are FSE encoded. The compressor builds optimal Huffman and FSE tables for this block based on frequency counts, then writes those tables into the block header followed by the entropy-coded bitstreams. If the combined size of tables plus bitstreams is larger than raw, the block would fall back to Raw.

Step 5: Frame Assembly. The block is wrapped in a Zstd frame. The frame header contains magic number (0xFD2FB528), frame content size (24), window size, dictionary ID (0 if none), and checksum flag. The block header specifies block type (Compressed) and block size. The compressed block payload is appended, followed by an optional XXH64 checksum of the original data for integrity. The final frame might be ~30 bytes for this tiny example (slight expansion due to table overhead), but for larger, more redundant data, the ratio improves dramatically—demonstrating why Zstd excels on larger inputs and with dictionaries for small ones.

5. Complexity Analysis

Zstandard's complexity is highly tunable, which is both its strength and a nuance for analysis. Unlike fixed algorithms like Huffman coding, Zstd's time and space depend heavily on the compression level, strategy, and input characteristics. Decompression, however, is intentionally simple and fast, with near-linear time regardless of how the data was compressed.

Compression time is dominated by match searching. At low levels (1-3), Zstd uses fast hash tables with limited chain search, giving near-linear time. At medium levels (4-9), it uses binary trees and hash chains with increasing search depth. At high levels (10-22), it employs optimal parsing that evaluates many possible parse paths using dynamic programming, which increases time significantly but yields better ratios. Memory usage during compression scales with window_log and hash_log, typically from 1 MB at level 1 to over 100 MB at level 22 with LDM enabled.

Decompression is much more predictable. It requires only a single pass over the compressed stream, decoding Huffman/FSE tables and copying matches via memmove. Its time is O(n) where n is the decompressed size, and its memory is O(w) where w is the window size (the maximum offset). This asymmetry—expensive compression, cheap decompression—is ideal for write-once, read-many workloads like web content distribution and database storage.

Where n is uncompressed size, w is window size (2^window_log), and s is total training sample size. Best case for compression occurs with highly redundant or incompressible data where match search terminates early. Worst case occurs with pathological data that forces exhaustive search without finding beneficial matches. In practice, Zstd's average compression throughput ranges from ~500 MB/s at level 1 to ~5-20 MB/s at level 19 on a modern x86 core, while decompression consistently achieves 1-3 GB/s.

6. Variations and Optimizations

Beyond the default levels, Zstandard offers a rich set of variations that tailor it to specialized workloads. Understanding these options allows engineers to extract maximum value without switching algorithms.

Negative Compression Levels: Introduced for ultra-fast scenarios, levels -1 to -22 (accessed via --fast=n in the CLI) sacrifice ratio for speed beyond level 1. They use an even more aggressive hash table with no chain search, achieving compression speeds exceeding 1 GB/s, faster than LZ4 in some cases, while still providing better ratio than LZ4. These are ideal for real-time telemetry or in-memory cache compression where CPU is the bottleneck.

Long Distance Matching (LDM): Enabled with --long, LDM uses a separate large hash table to find matches up to 128 MB away (or more with --windowLog). This is transformative for large files with distant redundancy, such as tar archives containing multiple versions of the same file, database backups, or genomic sequences. LDM adds modest memory overhead during compression but does not affect decompression memory beyond the window size.

Dictionary Variants: Zstd supports raw content dictionaries (just a byte blob) and structured dictionaries that include precomputed FSE/Huffman tables. Dictionaries can be trained with different selectors (e.g., cover algorithm) and can be made prefix dictionaries for streaming where each message is independent. There is also an experimental dictionary compressor that builds a dictionary optimized for a specific compression level.

Advanced Optimizations

Multi-threaded compression is natively supported. By splitting input into independent jobs (frames or blocks), Zstd can scale linearly across cores with --threads. Each thread compresses a chunk, and the output is a single concatenated frame that remains decompressible by single-threaded decoders. This is critical for compressing large backups quickly.

Another optimization is the use of prefetching and SIMD. The reference implementation uses explicit prefetch instructions and vectorized Huffman/FSE decoding to accelerate both compression and decompression. On modern CPUs, decompression is often memory-bandwidth bound rather than CPU bound.

Finally, Zstd offers a seekable format (via the separate zstd-seekable library). By storing an index of frame offsets and decompressed offsets, it allows random access to compressed data without decompressing the entire file, enabling efficient use in columnar storage like Parquet or for large log files where only a slice is needed.

7. Real-World Applications

Zstandard compression has seen explosive adoption since its standardization, replacing gzip and zlib in many performance-critical systems. Its versatility makes it suitable wherever data is stored or transmitted.

In web delivery, Zstd is now supported by all major browsers (Chrome, Firefox, Safari, Edge) as a Content-Encoding. CDNs like Cloudflare and Fastly use it to compress HTTP responses, achieving 10-15% smaller payloads than gzip at similar or faster speeds, directly improving page load times. Facebook itself uses Zstd to compress web assets, API responses, and even to compress data in transit between its data centers, saving petabytes of bandwidth daily.

Databases and storage systems are heavy users. MySQL, MariaDB, PostgreSQL, and RocksDB offer Zstd as a table or column compression option. RocksDB, also from Anthropic, uses Zstd by default for SST files, benefiting from its fast decompression during reads. The Linux kernel uses Zstd to compress the filesystem (Btrfs and SquashFS), kernel images, and hibernation images. Ubuntu and Fedora distribute packages compressed with Zstd, reducing install times. Backup tools like BorgBackup, Restic, and Facebook's MySQL backup system rely on Zstd for high-ratio, fast backups.

In data engineering, Apache Kafka, Hadoop, and Spark support Zstd for log and shuffle compression. Its dictionary mode is perfect for compressing small Kafka messages. Container ecosystems use Zstd for image layers: Docker and containerd support Zstd-compressed layers that decompress faster than gzip, speeding up container startup.

Networking protocols also benefit. Zstd is used in RPC frameworks like gRPC and Thrift for payload compression, and in data replication streams where low latency is critical. Its streaming API ensures that data can be compressed on the fly without buffering entire messages.

8. Comparison with Alternatives

Choosing a compression algorithm requires weighing ratio, speed, memory, and ecosystem support. Zstandard was explicitly designed to dominate the Pareto frontier, offering a better ratio than gzip at faster speeds, and faster decompression than most high-ratio compressors.

Gzip (DEFLATE) has been the de facto standard for decades, but it is limited to a 32 KB window and Huffman-only entropy coding, resulting in lower ratios and slower speeds compared to Zstd. LZ4 is the king of speed, with compression and decompression exceeding 3 GB/s, but its ratio is poor (often 30-50% worse than Zstd). Brotli, developed by Google, offers slightly better ratios than Zstd at maximum levels for web text, but its decompression is slower and it lacks Zstd's dictionary and LDM flexibility. xz (LZMA) achieves the highest ratios for archival but is extremely slow to compress and decompress.

The following table provides a feature-level comparison to guide selection:

In practice, the decision is often: use LZ4 when absolute lowest latency is required and ratio is secondary (e.g., in-memory caching), use Zstd for almost everything else where a balance is needed, and use xz only for cold archival where compression is one-time and decompression speed is irrelevant. For web text, Brotli at level 11 can edge out Zstd at level 19 in ratio by 1-2%, but Zstd at level 6 will compress 10x faster with only a small ratio penalty, making it preferable for dynamic content.

9. Common Mistakes and Best Practices

Despite its ease of use, teams often misuse Zstd in ways that negate its advantages or introduce subtle bugs. Avoiding these pitfalls ensures reliable, high-performance deployments.

Mistake 1: Using a Single Level for All Data. A common error is to hardcode level 3 everywhere. While level 3 is a good default, it is suboptimal for many workloads. Small, latency-sensitive RPCs should use level 1 or negative fast levels, while nightly backups should use level 19 with --long. Benchmark your actual data: compress a representative sample at levels 1, 3, 6, 9, and 19, and plot ratio vs. time to find the knee of the curve.

Mistake 2: Ignoring Dictionary Training for Small Data. Compressing thousands of 500-byte JSON records individually with Zstd without a dictionary yields poor ratios, often worse than gzip, because there is insufficient data to build good tables. The fix is to collect 100 MB of samples and train an 8-32 KB dictionary. Forgetting to ship the dictionary to decompressors is another failure mode—decompression will fail with "dictionary mismatch." Version your dictionaries and include the dict ID in your storage metadata.

Mistake 3: Not Bounding Decompression Output. Zstd frames store the decompressed size, but it is optional and can be spoofed. Decompressing untrusted input without a max_output_size limit exposes you to decompression bombs (e.g., a 10 KB payload that expands to 10 GB). Always use streaming decompression with a limit or pass max_output_size to decompress(). Similarly, failing to verify checksums (write_checksum=True) can hide data corruption.

Best Practices

• Reuse Contexts: Creating ZstdCompressor and ZstdDecompressor objects is expensive as they allocate window buffers. Reuse them across many compress/decompress calls rather than creating per-operation instances. They are thread-safe for independent operations if you create one per thread.
• Prefer Streaming for Large Data: Use stream_reader/stream_writer for files larger than a few MB to keep memory constant. One-shot compress() will allocate a buffer for the entire output.
• Enable Checksums for Storage: For data at rest, always enable checksums. The XXH64 checksum adds only 4 bytes per frame and catches bit rot and truncation.
• Measure Decompression, Not Just Compression: In read-heavy systems, decompression speed matters more. Zstd's decompression is level-independent, so you can often increase compression level for better ratio at no read cost.
• Use Concatenated Frames for Parallel Decompression: When compressing with multiple threads, the output is concatenated frames. Ensure your decompressor supports this (most do) and consider using the seekable format if you need random access.

Finally, do not recompress already compressed data (JPEG, MP4, encrypted blobs). Zstd will waste CPU and may even expand the data. Detect incompressible types and store them raw or with level 0 (no compression).

10. Interview Questions

Zstandard is a frequent topic in systems design and backend engineering interviews, especially for roles involving storage, networking, or performance. The following questions test both conceptual understanding and practical judgment.

Q1: Why does Zstd decompress much faster than it compresses, and why is this asymmetry desirable?

Compression must search for matches, build optimal entropy tables, and perform optimal parsing—an NP-hard-like search problem approximated with heuristics. This requires hash tables, binary trees, and dynamic programming. Decompression, in contrast, is a simple table lookup and memory copy: decode the FSE/Huffman codes to get literals and (offset, length) pairs, then copy bytes from the sliding window. No search is needed. This asymmetry is desirable because most data is written once and read many times (e.g., web assets, database SSTs, container images). Paying extra CPU once at write time to save storage and make every subsequent read faster is an excellent trade-off.

Q2: Explain how dictionary compression works in Zstd and when you would use it.

Dictionary compression trains on a corpus of similar small samples to build a shared dictionary containing common substrings and precomputed FSE/Huffman tables. When compressing a new small payload (e.g., 1 KB JSON), the compressor can reference strings in the dictionary as if they were part of the sliding window, and it starts with optimized entropy tables instead of building them from scratch. This can improve ratio by 2-4x on small data where normal Zstd would have no history. You would use it for Kafka messages, cache entries, or database rows where many small, schema-similar records are stored. Training requires >100 samples and the same dictionary must be available at decompression.

Q3: You need to compress a 500 GB database backup. How would you choose Zstd parameters, and how does it compare to gzip -9 and xz -6?

For a large backup, ratio and long-range redundancy matter more than real-time speed. I would use Zstd level 19 with --long (e.g., zstd -19 --long=30 --threads=0). --long enables 128 MB window to find duplicates across the entire dump, and level 19 uses optimal parsing. Multiple threads leverage all cores. Compared to gzip -9, Zstd -19 will achieve 10-20% better ratio and decompress 3-5x faster, though compression will be slower than gzip -9 but faster than xz. Compared to xz -6, Zstd -19 will have slightly worse ratio (xz may be 5% smaller) but will compress 5-10x faster and decompress 10x faster, making restores much quicker. For archival where restore time is critical, Zstd is usually preferred.

Q4: What is Finite State Entropy (FSE) and how does it improve upon Huffman coding used in DEFLATE?

FSE is an implementation of Asymmetric Numeral Systems (ANS). Huffman coding assigns an integer number of bits per symbol, which is inefficient for symbols with probabilities not a power of 1/2 (e.g., p=0.3 ideally needs ~1.74 bits, but Huffman gives 2). Arithmetic coding achieves fractional bits but is slow. FSE/ANS achieves near-arithmetic coding efficiency (fractional bits) with Huffman-like speed by using a finite state automaton and table-driven decoding. Zstd uses FSE for sequence codes and Huffman for literals, allowing it to encode data closer to its entropy limit than DEFLATE's Huffman-only approach, contributing to its better ratio at similar speeds.

Q5: How would you safely decompress untrusted Zstd data from a user upload?

Never use one-shot decompress() without limits. Use streaming decompression with a max output size and timeout. Example: create ZstdDecompressor(max_window_size=8*1024*1024) to limit memory, then use stream_reader with a loop that tracks total bytes written and aborts if it exceeds a threshold (e.g., 100 MB) or if time exceeds a limit. Enable checksum verification if the frame includes it, but do not trust the frame's declared content size. Also, handle concatenated frames and skippable frames correctly, and catch ZstdError exceptions for malformed input. This prevents decompression bombs and excessive memory allocation.