Polypress 1.0.0

Last updated 12 October 2026

Comparison by configuration

Size ratio by configuration

The size ratio is the size of the other output divided by the size of the Polypress archive. A ratio above 1 means that the Polypress archive is smaller. The pyarrow library could not read 100 of the tables, and ORC was not attempted on tables with more than 1,000 columns, so the columnar formats were compared on fewer tables.

ConfigurationTablesPolypress smallerMedian ratioLowest ratio
gzip -91,0001,0002.391.06
bzip2 -91,0009781.570.87
xz -9e1,0009901.380.82
zstd -31,0001,0002.271.13
zstd -191,0009971.490.88
zstd --ultra -221,0009971.480.88
brotli -q 111,0009781.400.79
lz4 -91,0001,0002.861.33
zip -91,0001,0002.401.08
7-Zip, LZMA2, level 91,0009921.450.84
7-Zip, PPMd, order 61,0008961.400.79
7-Zip, PPMd, order 121,0008211.230.75
7-Zip, PPMd, order 161,0008091.200.75
Parquet, snappy9009002.681.04
Parquet, gzip level 99008991.990.80
Parquet, brotli level 119008991.740.67
Parquet, zstd level 229008991.830.73
ORC, snappy8778773.171.09
ORC, zlib8778752.200.71
ORC, zstd8778722.080.65
Feather, lz49009005.851.47
Feather, zstd9009003.561.18

Effect of the final-stage compressor

Polypress compresses its modelled streams with LZMA2, the method used by xz. Parquet does not offer LZMA2. To separate the effect of the modelling from the choice of final-stage compressor, the modelled streams were also compressed with zstd at level 22 and with brotli at level 11, and compared with outputs that use the same compressor at the same level.

Polypress streamsCompared withPolypress smallerMedian ratio
zstd, level 22Parquet, zstd level 22899 of 9001.65
zstd, level 22zstd --ultra -22984 of 1,0001.35
brotli, level 11Parquet, brotli level 11899 of 9001.72
brotli, level 11brotli -q 11988 of 1,0001.38

Tables on which another configuration was smaller

Another configuration produced a smaller output on 219 tables: 7-Zip with PPMd on 206, brotli on 6, ORC on 5, and xz and bzip2 on 1 each. PPMd is a context-modelling method designed for text. The largest difference was on a table of SARS-CoV-2 variant proportions, where the output of ORC with zstd was 35% smaller than the Polypress archive.

Fidelity of text in Parquet files

The pyarrow library inferred the column types when it read each CSV file. Converting the resulting Parquet file back to text reproduced every cell for 172 tables and changed at least one cell for 702. The remaining 126 could not be checked. A typical change is 1.50 becoming 1.5. For these tables, the Parquet output is smaller partly because it discards the original formatting, which favours Parquet in the comparison.