Aster TSDB, Part 3: Inside the Storage Engine — Time-Aware Compaction and 1.7 Bytes per Row
How Aster TSDB stores 20 million industrial rows in 33 MB instead of 238, and why compressing them made every query we measured the same or faster.
In Part 1 we introduced Aster TSDB, our open-source time-series database built on Apache Cassandra. In Part 2 we covered the query side: 21 native functions, gap-fill and full-text search. This final part goes underneath all of that, into the storage engine.
It is the most technical of the three, and the part we learned the most from. Every design here started as a response to a specific operational pain we had with time-series data on stock Cassandra:
- Retention by TTL creates tombstones, and clearing tombstones takes compaction I/O that competes with writes.
- Range reads touch many SSTables per partition, because nothing keeps a time range in one file.
- A few rows of late data from a reconnecting edge device can pull a large, old SSTable back into compaction.
- Row storage repeats the clustering key, timestamp and cell headers for every sample, and general-purpose compression only partly removes that.
Aster addresses these at three layers: a compaction strategy that organizes SSTables by time, a memtable that knows about time windows from the moment a row is written, and a tiered storage engine that re-encodes closed windows into a purpose-built columnar format. Here is how each works and why it is built the way it is.
Compaction that understands time
Cassandra’s general-purpose compaction strategies do not know what time it is. That is the root of most operational pain with time-series data on Cassandra: tombstones from TTLs, read amplification across many SSTables, and late data dragging old files back into compaction.
Aster adds TimeSeriesCompactionStrategy, or TSCS. It combines TWCS’s time ordering and whole-window drops with UCS’s in-window compaction, and adds several things neither has. SSTables are classified into fixed time windows, and each window goes through a well-defined life:
- Current and recently closed windows are delegated to a separate UnifiedCompactionStrategy instance per window. Inside a window, you get UCS’s write-optimized behaviour unchanged. Across windows, SSTables are never mixed in one compaction.
- After freeze_after, a closed window is compacted down to exactly one SSTable and then left alone. A partition-plus-time-range query touches one file per window, so read amplification is minimal. Data that has already expired at the moment of freezing is reclaimed during that final compaction.
- After retention, a window’s SSTable is simply deleted. No compaction, no tombstones, no I/O spike.
- Late and backfilled data is split at window boundaries during flush and streaming, so three days of backlog from a reconnecting edge device lands locally in the three past windows it belongs to, and only those windows are touched. Legacy SSTables that span windows are split automatically.
- A far-future guard (max_future_window) keeps a device with a broken clock from polluting windows years ahead.
- A virtual table, system_views.timeseries_windows, shows each window’s state, the SSTables that span it and its row count, so you can see what compaction is doing instead of guessing.
ALTER TABLE pp.tm_tag_point WITH compaction = {
'class': 'TimeSeriesCompactionStrategy',
'window_size': '1d',
'freeze_after': '2h',
'scaling_parameters': 'T4',
'retention': '62d'
};A memtable that knows about windows
Running TSCS in production taught us something the design had not anticipated. Window splitting happened at flush time: to cut a partition at window boundaries, the flush path buffered the whole partition in heap and re-routed its rows. For partitions larger than 64 MiB, it gave up and wrote the partition whole, which produced an SSTable spanning windows. Because its shape never changed, the split-and-refreeze cycle could never converge, and that window got stuck. We call it “parked.” We saw this on a production node, not in a test.
The fix was to stop doing the work at flush time at all. TimeSeriesMemtable is an opt-in, per-table memtable that assigns each row to its TSCS window at the moment it is written, putting it into that window’s shard in O(1). There is no partition buffering at flush, so the 64 MiB cap disappears. Rows are stored in primitive-array columns rather than per-cell objects, which cut heap usage per row by 5.5× in our measurements. Reads binary-search slices and assemble only the rows a query needs, streaming them out without holding anything. On a tiered table, cold windows flush straight into the compressed chunk format.
Tiered storage and the chunk format behind the 7.1×
This is where the storage savings come from.
Turn on a timeseries_tiering policy on a table, and a background re-encoder starts working through windows that are older than hot_window. Each such window is rewritten into a columnar chunk and stored in a shadow table named <table>__chunks, and the original rows are removed. Recent data stays as ordinary rows, so the write path and live reads are untouched. Turning it on is one line of CQL:
ALTER TABLE pp.tm_tag_point WITH extensions = {
'timeseries_tiering':
'{"hot_window":"2d","chunk_window":"1d","cold_window":"3650d","interval":"1h"}'
};-- check the policy and run statistics
SELECT * FROM system_views.timeseries_tiering;The re-encoder then runs on the interval schedule, or on demand with nodetool retier. nodetool tieringstatus shows the policy, the last run and cumulative statistics per table. cold_window acts as retention for compressed data: chunks older than that are dropped whole.
Why columnar chunks beat Zstd
Row storage in Cassandra is already compressed. On our production node, the tag table gets a Zstd ratio of 0.19, about 5.3×. So why does the chunk format win so decisively on top of that?
Get 이상부’s stories in your inbox
Join Medium for free to get updates from this writer.
Not because we wrote a better general-purpose compressor. Row serialization repeats the clustering key, the timestamp and per-cell headers for every single row, and a general-purpose compressor can only partially see through that repetition. A column-oriented encoding removes the repetition from the representation itself: the timestamp axis is stored once for all columns, each column is stored contiguously, and each column gets an encoding chosen for its type and its actual values. Then the SSTable compression layer runs on top of that, as before.
Chunk format v4
One chunk holds one partition for one window: a shared timestamp axis plus an independent section for every regular column. Each column is cut into 1,024-row blocks. For every block, the encoder tries every applicable encoding and keeps whichever is actually smallest, so the format adapts block by block rather than committing to one codec per column. Every block is independently decodable and randomly addressable.
- double: ALP and ALP-RD.
- Integers and timestamps: frame-of-reference and delta bit-packing. A regularly spaced timestamp axis has a constant delta, so the packed width is zero bits; it costs about 0.02 bytes per row, which is just the fixed block-table overhead. A single missing sample does not widen the packing for the whole block; it goes into a small exception area.
- boolean: one bit per value.
- text and opaque bytes: dictionary (DICT) or RAW.
- A column whose value never changes in the window (quality = 192, error_code = 0, an empty attribute map): CONSTANT. The value is written once in the directory, and the column costs zero bytes per row.
- A column nobody writes (value_boolean on a numeric tag): ALL_NULL. One flag, zero bytes per row.
- Nulls inside a block use one of four presence modes, chosen per block.
On the production table shape, four of the eight regular columns cost zero bytes per row. That, plus ALP on the sensor values and a near-free timestamp axis, is how a row ends up at about 1.7 bytes.
Why ALP, and why only ALP
For floating-point sensor values we ran a proper bake-off, first Gorilla against Chimp128, then the winner against ALP, on distributions taken from our real workload. On the production shape — readings quantized to two decimal places — ALP came in at 0.752 bytes per value against Chimp128’s 1.351, 0.56× the size. On integer-valued walks it was 0.40×. On near-constant series it was 0.011×.
ALP is not universally better. On full-precision, smoothly varying doubles that have never been rounded, Chimp128 is 1.5–3.9% smaller, because it exploits sequential correlation by XOR-ing each value against a recent similar one, and ALP-RD only models the static distribution of bit patterns. We accepted that loss deliberately. Those shapes do not exist in our measured production data, where every reading is decimal-quantized. And Chimp’s XOR chain is inherently serial, which forces decoding a whole chunk from the start. ALP is block-based, which is what makes independent block decode and random access possible. So ALP and ALP-RD are the only double codec in the tree; Gorilla and Chimp128 were removed. There is no codec option to tune, because there is no wrong choice left to make.
Design rules we wrote down
A few decisions are recorded in the format specification explicitly, so nobody quietly undoes them later:
- Statistics in a chunk are for pruning only: the chunk’s time bounds and per-column and per-block min/max. No sums, sketches or histograms. Pruning lets a query skip chunks and blocks it does not need; anything more belongs in a separate design.
- The column directory stores explicit section offsets, so a query that projects two columns jumps straight to those two sections and never touches the others.
- The encoder is byte-deterministic. The same input always produces the same bytes, regardless of JIT tier — we verify this in a forked JVM with -XX:TieredStopAtLevel=1. That makes “has this chunk changed?” checks and replica comparisons safe.
- Formats v1 to v3 were removed rather than kept behind a dual read path. Reading one fails loudly with UnsupportedChunkFormatException instead of silently returning something partial.
Transparent reads: the application never knows
None of this is visible to your application. An ordinary SELECT on the base table automatically merges hot rows with decoded chunk rows. Partition-plus-time-range queries, point lookups, avg, count and time_bucket aggregates, gap-fill, LIMIT, ORDER BY DESC and paging all work across the boundary between hot and cold. A query that starts above the cold boundary skips the merge and returns the ordinary hot iterator, at no extra cost. Late data that arrives for an already-chunked window is merged into the chunk on the next cycle.
Safety rails
Tiering deletes base rows after encoding them, so we were careful about where it can go wrong. The re-encoder only accepts quorum-strength consistency levels; anything weaker is rejected, because a write that looked successful but did not reach a quorum would lose data outright. Invalid combinations of hot_window, chunk_window and cold_window are refused at ALTER time. Tables that are not on TSCS are refused, as are indexes and views on regular columns that tiering could make inconsistent. Settings that are legal but probably not what you meant, such as a TTL shorter than hot_window so nothing is ever compressed, produce a WARN that names both values. The repository includes a production rollout checklist that walks through the irreversible points before you turn tiering on for a real table.
What’s next
Profiling the chunk scan showed something useful: the ALP decode and bit-unpacking arithmetic is only 5–8% of the time. About 92% is cursor movement and row assembly. So the next gains are not mainly about SIMD-accelerating the codecs; they are about not materializing rows at all. On the roadmap: compressed query execution that aggregates directly over encoded chunks, vectorized aggregation kernels, SIMD decode where it does pay off, and continuous aggregates for time-bucket rollups.
Summary
Taken together, the storage engine follows one idea: time is the primary organizing principle of this data, so every layer should know about it. The memtable assigns a row to its window when it is written. Compaction keeps windows apart, freezes them when they close and drops them whole when they expire. Tiering re-encodes closed windows into columns, with an encoding chosen block by block, and reads merge them back so transparently that the application never notices — except that the disk is 7.1× smaller and the queries are faster.
That is the end of the series. Part 1 covers the overview and benchmarks; Part 2 covers the CQL functions, gap-fill and full-text search. The format specification, the codec bake-off, the benchmarks with their reproduction steps, and the production rollout checklist are all in the repository. If you run time series on Cassandra, try tiering on a copy of one of your tables and tell us what you measure. Issues, pull requests and stars are all very welcome.
