Last updated 12 October 2026
Corpus and configurations
Selection of tables
Datasets were selected from the Socrata catalogue in descending order of page views. A dataset was included if its CSV export could be downloaded, it had at least two columns and 20 rows, it was at least 51,200 bytes (50 KiB) in size, and it was not a byte-for-byte copy of a dataset already included. Only the first 50,000 rows of each dataset were downloaded. Of 1,184 datasets considered, 1,000 met these criteria.
Tables larger than 16 MB (189 of the 1,000) were truncated at the last complete row before 16 MB. Every configuration received the same truncated file.
The corpus
- Tables
- 1,000
- Total size as CSV, after truncation
- 7.16 GB
- Rows
- 27.9 million
- Columns
- 22,132
Configurations compared
| Group | Configurations |
|---|---|
| General-purpose compressors | gzip -9, bzip2 -9, xz -9e,
zstd -3, zstd -19, zstd --ultra -22, brotli -q 11,
lz4 -9 |
| Archivers | zip -9; 7-Zip with LZMA2 at level 9; 7-Zip with PPMd at model
orders 6, 12 and 16 |
| Columnar formats, written with pyarrow | Parquet with snappy, gzip (level 9), brotli (level 11) and zstd (level 22); ORC with snappy, zlib and zstd at default levels; Feather with lz4 and zstd |