Polypress 1.0.0

Last updated 12 October 2026

Corpus and configurations

Selection of tables

Datasets were selected from the Socrata catalogue in descending order of page views. A dataset was included if its CSV export could be downloaded, it had at least two columns and 20 rows, it was at least 51,200 bytes (50 KiB) in size, and it was not a byte-for-byte copy of a dataset already included. Only the first 50,000 rows of each dataset were downloaded. Of 1,184 datasets considered, 1,000 met these criteria.

Tables larger than 16 MB (189 of the 1,000) were truncated at the last complete row before 16 MB. Every configuration received the same truncated file.

The corpus

Tables
1,000
Total size as CSV, after truncation
7.16 GB
Rows
27.9 million
Columns
22,132

Configurations compared

GroupConfigurations
General-purpose compressorsgzip -9, bzip2 -9, xz -9e, zstd -3, zstd -19, zstd --ultra -22, brotli -q 11, lz4 -9
Archiverszip -9; 7-Zip with LZMA2 at level 9; 7-Zip with PPMd at model orders 6, 12 and 16
Columnar formats, written with pyarrowParquet with snappy, gzip (level 9), brotli (level 11) and zstd (level 22); ORC with snappy, zlib and zstd at default levels; Feather with lz4 and zstd