A table is not
a stream of bytes.

1. Predict down a column

   Fit a low-degree polynomial to the last few values in a column, extrapolate one step, store only the error. Because the fit is re-centred at every cell, the coefficients never need to be stored — the decoder already has the neighbours. Order-k extrapolation turns out to be exactly the k-th finite difference, so this is one np.diff call.

2. The same thing in two dimensions

   Where adjacent numeric columns are commensurable — same decimal places, same magnitude — a cell is predicted from its left, upper, and upper-left neighbours. This is what wins on matrix-shaped tables, and it is the one thing no shipped columnar codec does: Gorilla, DoubleDelta, T64, zfp and fpzip all predict down a single column. The detection matters as much as the predictor — differencing Model Year against Make is meaningless, so groups form only where columns are genuinely comparable.

3. Reorder the rows so a column collapses into runs

   This is the central idea. Rather than model the fact that City depends on Postal Code, sort the rows by the parent column: equal parents become adjacent, the child collapses into long runs, and the entropy coder eats it. Parents are picked by conditional entropy, then confirmed by actually compressing both ways, and arranged into a tree so a parent is always decoded before its children.

Counts differ because pyarrow could not read every CSV: 50 files defeated its Parquet/Feather reader, 64 its ORC writer.

It loses on 22 of 500, and all 22 are listed publicly

Twelve losses are to brotli -q 11, which is not one of the carried fallbacks. Nine are to bzip2 -9 run on the original file — Polypress's fallback compresses the table re-rendered canonically, and on those tables normalising the quoting removed redundancy the Burrows–Wheeler transform had been exploiting; the gap is 0.1%–1.6%. That leaves one genuine loss, sars_cov_2_variant_proportions at 0.65× to orc+zstd, and it is undiagnosed.

The honest form of the guarantee is never worse than our own plain fallback — not “never worse than any tool on your original bytes.”

• The parent search is O(columns²)

  A 209-column table means 38,220 scored pairs. The worst case for speed is not a big file — it is a 60 KB table with 106 columns and 79 rows, at 0.5 MB/s.

• The 2× results are matrix-shaped tables

  1.3–1.5× is the typical result.

• Every benchmark dataset is a government open-data table

  That is what the Socrata catalog indexes. Census microdata, NOAA grids, genomics and financial tick data are unmeasured, and there is no reason to assume the result transfers.

• Nobody outside this project has run it

  No independent replication of any number on this page.

• The competitor set had a hole

  All 17 were LZ-family, Huffman or block-sort — no context-mixing model. PPMd was added later and Polypress still wins 13/13 against it, but on the seven tables where row reordering does not fire, PPMd gets within 1–5%. Sweeps run before 2026-08-06 should be quoted as “17 LZ-family and columnar competitors”.

• The Mac app is unsigned

  Gatekeeper warns on first open; right-click → Open clears it.

# the codec and the `polypress` command
pip install polypress
# add pyarrow, for .parquet in and out
pip install 'polypress[parquet]'

polypress compress data.csv                      # -> data.csv.ppz
polypress restore  data.csv.ppz                  # -> data.csv
polypress restore  data.csv.ppz -o out.parquet   # doubles as a converter
polypress info     data.csv.ppz                  # plan, shape, how much was reordered

Version 0.2.0, MIT. Python 3.9+ and numpy; a C compiler is optional — the accelerator builds itself on first import and falls back to numpy without one. The source is public, so git clone then pip install . from work/ also works, and python3 work/tzip.py runs out of the checkout with nothing installed.