Columnar Compression Strategies to Accelerate ML Pipelines and Cut Storage Costs

Why columnar compression matters for ML pipelines

Many data science pipelines spend more time moving data than computing gradients. Columnar compression can reduce I/O, lower storage costs and speed up training when the pipeline is I/O bound. This article walks through practical strategies for compressing columnar data, trade offs to consider, and concrete implementation patterns using Python libraries commonly used in production.

Big picture: what changes with columnar storage

Columnar formats store each field together, which enables three useful effects for ML workflows:

• Selective reads so you only load features required by a model or by a transformation step.
• Better compression because similar values are colocated and compression schemes like dictionary or run length encoding become effective.
• Efficient predicate pushdown where file scanners skip row groups that do not match filters, reducing disk reads.

Compression techniques and when to use them

Not every codec or encoding is optimal for every column. Consider these patterns.

• Codec compression such as ZSTD, SNAPPY, GZIP. ZSTD often yields a good size vs CPU trade off. SNAPPY decodes faster but compresses less.
• Dictionary encoding effective for low cardinality strings and categorical fields. It turns repeated values into small integer codes.
• Run length encoding (RLE) good when values repeat in long sequences, for example time series with long stretches of identical labels.
• Delta encoding for monotonically increasing numeric columns like timestamps or sorted ids. It stores differences instead of full values.
• Quantization for floating point features where a small precision loss is acceptable to save space, useful for models resilient to small noise.

File formats and their strengths

• Parquet — widely adopted in the Python ecosystem, supports column-level encodings, row groups, and predicate pushdown.
• ORC — similar benefits, often used in Hadoop environments.
• Feather/IPC — very fast for single-machine round trips, less mature for large scale partitioned storage.

Parquet often becomes the default for data lakes. The rest of examples focus on Parquet and pyarrow because they combine flexibility and solid Python support.

Concrete example: writing compressed Parquet with pyarrow

The snippet below shows how to apply dictionary encoding to categorical columns and use ZSTD compression for a Parquet file. Dictionary encoding is set at the column level, and ZSTD is selected for overall compression.

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# sample dataframe
df = pd.DataFrame({
    'user_id': range(100000),
    'country': ['US','CA','US','DE'] * 25000,
    'event_time': pd.date_range(start='2021-01-01', periods=100000, freq='T'),
    'value': (pd.np.random.rand(100000) * 100).round(3)
})

# cast low cardinality strings to category
df['country'] = df['country'].astype('category')

# convert to Arrow table
table = pa.Table.from_pandas(df, preserve_index=False)

# write parquet with ZSTD and dictionary encoding for category column
pq.write_table(
    table,
    'events.zstd.parquet',
    compression='ZSTD',
    use_dictionary=['country'],
    flavor='spark'
)

Notes about the example: dictionary encoding helps heavily when many repeats exist. ZSTD can reduce storage substantially with reasonable CPU cost. The flavor option can affect interop with some engines.

Reading selectively and leveraging predicate pushdown

When training models you rarely need every column or every row. Use column projection and row filtering at read time to avoid unnecessary I/O. The following shows a dataset scan using pyarrow dataset APIs.

import pyarrow.dataset as ds

dataset = ds.dataset('events.zstd.parquet', format='parquet')

# read only features needed and apply a date filter
scanner = ds.Scanner.from_dataset(
    dataset,
    columns=['user_id','country','value'],
    filter=ds.field('event_time') >= pa.scalar('2021-01-02')
)

table = scanner.to_table()
df_subset = table.to_pandas()

Using the dataset API allows the format reader to skip row groups that do not match the filter and to load only requested columns. This is often faster than reading the whole file and then subsetting in memory.

Partitioning and layout strategies

Layout matters. Partitioning files by high-cardinality fields can backfire, but partitioning by date or coarse buckets often helps. Consider a combination:

• Partition by date for time based workloads, for example year/month/day.
• Bucket or hash-partition heavy cardinality keys only when queries frequently filter by those keys.
• Keep row group size moderate, for example tens to hundreds of megabytes, so skipping row groups is efficient without creating too many tiny files.

In a production lake, avoid millions of tiny files. Also remember that too much partitioning can increase metadata overhead and slow down file discovery.

Integrating into ML pipelines

How does this mix into training? A few pragmatic patterns:

• Feature stores can materialize features as partitioned Parquet with dictionary encoding for categorical features so downstream jobs read only needed feature columns.
• Batch training can use dataset scanners and chunked reads to stream minibatches into a trainer or to feed incremental learners like SGDClassifier via partial_fit.
• Distributed training with Dask or Spark works well when files are partitioned sensibly and compression balances network transfer time.
• Memory mapping and in-process columnar engines like Polars can reduce copies when transforming features just before model input.

Example: streaming parquet into an incremental sklearn training loop. This pattern avoids loading the entire dataset into RAM.

from sklearn.linear_model import SGDRegressor
import pandas as pd
import pyarrow.dataset as ds

dataset = ds.dataset('events.zstd.parquet', format='parquet')
model = SGDRegressor(max_iter=1, tol=None)

for batch in dataset.to_table().to_batches(batch_size=10000):
    df_batch = batch.to_pandas()
    X = df_batch[['value']].values
    y = df_batch['user_id'].values % 100  # placeholder target for example
    model.partial_fit(X, y)

Practical checklist before you compress

• Profile I/O to confirm the workload is I/O bound. Compression helps most when reads/writes dominate CPU.
• Identify columns with low cardinality and mark them for dictionary encoding.
• Test codecs on representative samples. Measure compressed size and read throughput rather than assuming best default.
• Consider read-time CPU budget: very heavy compression can increase CPU during decode, which matters on shared clusters.
• Automate schema and encoding decisions in ingestion pipelines so downstream steps can rely on consistent layout.

Small experiments on a sample of your data usually reveal the best trade off between storage and decode time. Keep a simple benchmark script that measures storage size, write time and read throughput under realistic concurrency.

Tools that accelerate adoption

• pyarrow for flexible Parquet writes and dataset scanning.
• fastparquet as an alternative Parquet engine in some pandas workflows.
• Polars when you need very fast single-node columnar operations.
• Dask for distributed reads and parallel preprocessing before training.

Combine tools: use Dask or Polars to transform and partition data, then write Parquet with pyarrow and appropriate compression. Keep the code that chooses encodings as part of the ETL so experiments remain reproducible.

Final notes and next steps

Columnar compression can speed up pipelines and lower costs, but benefits depend on data characteristics and workload patterns. Start with profiling, apply dictionary encoding to categorical fields, choose a codec that fits your CPU budget, and design partitions around common filters. With these measures, reading only the needed features and skipping irrelevant rows becomes practical, which often shortens iteration cycles and reduces storage spend.