<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>CROWDB Blog</title><subtitle>Notes from inside a distributed storage system</subtitle>
<link href="https://giancao.github.io/feed.xml" rel="self"/><link href="https://giancao.github.io/"/>
<id>https://giancao.github.io/</id><updated>2026-11-08T23:16:07+00:00</updated><author><name>Gian Crow</name></author>
<entry><title>Raft + RocksDB to CrowDB: A Journal–Tree Recovery Contract</title><link href="https://giancao.github.io/blog/raft-rocksdb-disable-wal/"/><id>https://giancao.github.io/blog/raft-rocksdb-disable-wal/</id><published>2026-10-08T04:00:00+00:00</published><updated>2026-10-08T04:00:00+00:00</updated><summary>The journal preserves recent ordered history; the tree materializes a durable contiguous prefix.</summary><content type="html">&lt;h2 id=&quot;start-with-the-common-layout&quot;&gt;Start with the common layout&lt;/h2&gt;

&lt;p&gt;A common distributed KV store puts a replicated consensus layer in front of a local storage engine. A write follows roughly this path:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/distributed-kv-write-path.png&quot; alt=&quot;Common distributed KV write path from consensus journal to local engine and snapshot&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Consensus orders a mutation; the local engine materializes it.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The consensus journal establishes the order that replicas must agree on. The local engine WAL protects an applied command while its data is still in memory or waiting for a flush. They protect different durability boundaries, but they can record the same mutation twice.&lt;/p&gt;

&lt;p&gt;This layout appears in established systems such as &lt;a href=&quot;https://tikv.org/docs/7.1/reference/architecture/overview/&quot;&gt;TiKV&lt;/a&gt; and &lt;a href=&quot;https://docs.yugabyte.com/stable/architecture/docdb/&quot;&gt;YugabyteDB&lt;/a&gt;. Their production experience deserves respect; we use them here only as architectural context. Their implementations differ, but the boundary is familiar: a replicated journal drives a local LSM-based engine.&lt;/p&gt;

&lt;p&gt;For a general-purpose distributed KV store, Raft plus RocksDB is a sensible baseline. Raft supplies replication and recovery order; RocksDB supplies compaction, indexing, and local persistence. The difficult part is the boundary between them: when is a command applied, when is its state durable, and when can the journal record be reclaimed?&lt;/p&gt;

&lt;h2 id=&quot;the-hidden-recovery-boundary&quot;&gt;The hidden recovery boundary&lt;/h2&gt;

&lt;p&gt;This is the double-journal problem. RocksDB can disable its WAL when the outer consensus journal is authoritative. The benefit is less duplicate logging. The cost is that the system must connect two kinds of progress:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;the journal knows which operations have been ordered and committed;&lt;/li&gt;
  &lt;li&gt;the storage engine knows which materialized state has reached durable storage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If those progress markers diverge, recovery can skip a mutation that the tree still needs. YugabyteDB provides a useful documented example: DocDB disables the RocksDB WAL and tracks how much Raft history has reached durable SST files. That boundary determines which Raft records can be replayed and which can be reclaimed. (&lt;a href=&quot;https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log-%28WAL%29&quot;&gt;RocksDB WAL documentation&lt;/a&gt;, &lt;a href=&quot;https://docs.yugabyte.com/stable/architecture/docdb/performance/&quot;&gt;YugabyteDB’s design&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;The important idea is larger than &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;disableWAL&lt;/code&gt;. The journal preserves recent ordered history; the tree materializes a durable prefix. The recovery boundary between them must be explicit.&lt;/p&gt;

&lt;h2 id=&quot;the-slot-is-the-journal-position-contract&quot;&gt;The slot is the journal position contract&lt;/h2&gt;

&lt;p&gt;CrowDB makes that boundary a storage-engine contract. Its lower KV layer is a Multi-Paxos KV store. Each replica’s Paxos WAL records values accepted for slots. When the upper layer applies a selected slot, it passes that slot position to the tree, which materializes the mutation. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;crowdb-tree&lt;/code&gt; does not add a separate mutation redo WAL.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/journal-position-contract.png&quot; alt=&quot;CrowDB journal position contract from Paxos journal through L0 and L1 to a persistent snapshot&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The journal position connects ordered history to the durable tree frontier.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The journal does not need to tell the tree who the leader was, which ballot was used, or how the quorum was formed. The tree needs a mutation and its ordered position:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;mutation(key, value_or_tombstone, slot)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;It can then report the frontier through which its materialized state is durable. The journal and the tree are two parts of one recovery system: the journal keeps recent history, and the tree keeps the durable materialized prefix.&lt;/p&gt;

&lt;p&gt;The slot is also the replacement boundary. A different journal can feed the same tree as long as it preserves the position protocol: it must identify mutations by position, allow positions to form a contiguous prefix, replay records after the durable frontier, and reclaim old records only after the tree has published that prefix. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;slot&lt;/code&gt; is the name used by the current Paxos implementation; the contract is more general than Paxos.&lt;/p&gt;

&lt;p&gt;The tree stores the position in the value cell rather than appending it to the sorted key:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;key -&amp;gt; (resolved_slot, value_or_tombstone)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;CrowDB keeps one current version per key. If &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;account/status&lt;/code&gt; already contains slot 104, a delayed application of slot 101 cannot replace it. A tombstone carries its slot as well, so an older put cannot resurrect a key that a newer deletion still protects. The tree does not need to understand leaders, ballots, or network messages.&lt;/p&gt;

&lt;h2 id=&quot;execution-can-be-out-of-order-durability-cannot&quot;&gt;Execution can be out of order; durability cannot&lt;/h2&gt;

&lt;p&gt;CrowDB’s Paxos group admits multiple in-flight slots within a configured window. The L0 MemTable can accept non-contiguous slots, so a missing slot does not stop later applications from making progress in memory.&lt;/p&gt;

&lt;p&gt;For example, slots 101, 103, and 104 may already be applied while 102 is missing:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/out-of-order-contiguous-frontier.png&quot; alt=&quot;Out-of-order L0 execution and contiguous L1 durability frontier&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;L0 accepts a window of non-contiguous slots; L1 publishes only the contiguous prefix.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Only the flush from L0 into the L1 tree publishes that contiguous prefix. Publishing 104 while 102 is missing would let recovery discard a journal record that the tree still needs. This keeps the Paxos group concurrent while keeping the durable recovery frontier simple. We will discuss the reasons for choosing Paxos separately.&lt;/p&gt;

&lt;p&gt;The tree code implements this boundary directly. It tracks received slots, computes the contiguous slot frontier, and leaves L0 entries above the flush frontier for a later flush. A highest observed slot is not the same thing as a safe durable slot.&lt;/p&gt;

&lt;h2 id=&quot;recovery-snapshots-and-journal-reclamation&quot;&gt;Recovery, snapshots, and journal reclamation&lt;/h2&gt;

&lt;p&gt;Consider a crash where the consensus layer has committed through slot 100, the state machine has applied through slot 100 in memory, and the durable tree covers only slot 90.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/raft-recovery-frontier.png&quot; alt=&quot;Recovery frontier after a crash: committed log, applied state, durable checkpoint, and required replay&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The tree checkpoint at 90 is durable; slots 91–100 remain in the journal for replay.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Removing the state-machine WAL is safe only while the journal preserves that missing suffix. If an integration stores data and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;applied_index&lt;/code&gt; independently, it can flush the marker first, crash, and later read “100” from state that only contains 90. The error is in the integration contract, not in RocksDB.&lt;/p&gt;

&lt;p&gt;A snapshot therefore records more than a root identifier. It also records the frontier it covers. Recovery loads the tree at that frontier and replays journal records after it. The same frontier tells journal maintenance which older records are safe to reclaim. Flush is not merely “write a larger SST”; it publishes a contiguous part of the journal as durable tree state.&lt;/p&gt;

&lt;p&gt;RocksDB provides mechanisms such as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;atomic_flush&lt;/code&gt; to coordinate flushes across column families. The enclosing system must still define which flush boundary corresponds to which journal position, which snapshot is recoverable, and which records remain necessary. (&lt;a href=&quot;https://github.com/facebook/rocksdb/wiki/Atomic-flush&quot;&gt;Atomic flush&lt;/a&gt;)&lt;/p&gt;

&lt;h2 id=&quot;the-same-contract-can-serve-another-journal&quot;&gt;The same contract can serve another journal&lt;/h2&gt;

&lt;p&gt;CrowDB’s upper Chunk-KV layer has its own partition sequencer and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PartitionJournal&lt;/code&gt;. That journal stores records through ChunkStream, while the tree uses a chunk-backed page store. It does not reuse the lower Paxos journal implementation or treat a stream byte offset as a Paxos slot.&lt;/p&gt;

&lt;p&gt;It does reuse the tree’s progress-aware materialization interface. The adapter passes &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;mutation_seq&lt;/code&gt; into tree apply operations, and a condition-failed record advances the corresponding no-op progress. The tree can therefore serve different ordering and journal implementations as long as they provide the same position and frontier semantics.&lt;/p&gt;

&lt;p&gt;CrowDB owns this flow across journal, apply, tree, snapshot, and reclamation. That makes it possible to change one part while keeping the recovery protocol visible and testable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;In CrowDB, the journal and the tree are not two independent persistence layers. The journal preserves recent ordered history; the tree materializes a durable contiguous prefix. A shared slot frontier connects them into one recovery system.&lt;/strong&gt;&lt;/p&gt;
</content></entry><entry><title>Load TPC-H and TPC-DS tables into CROWDB Iceberg</title><link href="https://giancao.github.io/blog/load-tpc-h-and-tpc-ds-into-crowdb-iceberg/"/><id>https://giancao.github.io/blog/load-tpc-h-and-tpc-ds-into-crowdb-iceberg/</id><published>2026-09-30T15:20:00+00:00</published><updated>2026-10-06T08:30:00+00:00</updated><summary>Load 32 Iceberg tables at SF1, check their row counts, and read both datasets with DuckDB.</summary><content type="html">&lt;p&gt;The &lt;a href=&quot;https://hub.docker.com/r/crowdb/crowdb-iceberg/tags&quot;&gt;CROWDB Iceberg latest container&lt;/a&gt; serves a catalog, the files behind its tables, and a local web Console. This guide loads TPC-H and TPC-DS datasets at scale factor 1 (SF1), checks the import reports, and reads a table with DuckDB.&lt;/p&gt;

&lt;p&gt;Use disposable data on a Linux amd64 host with Docker, Python 3.10–3.12, the DuckDB CLI, and a free local port 9092. The loader may download a TPC-H generator on its first run; DuckDB may download extensions.&lt;/p&gt;

&lt;h2 id=&quot;1-start-crowdb-iceberg&quot;&gt;1. Start CROWDB Iceberg&lt;/h2&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;docker run &lt;span class=&quot;nt&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;--name&lt;/span&gt; crowdb-iceberg &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;-p&lt;/span&gt; 127.0.0.1:9090:9090 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;-p&lt;/span&gt; 127.0.0.1:9092:9092 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  crowdb/crowdb-iceberg:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Open &lt;a href=&quot;http://127.0.0.1:9090/&quot;&gt;http://127.0.0.1:9090/&lt;/a&gt; in a browser and select &lt;strong&gt;Iceberg&lt;/strong&gt;. Leave the Console open while the loader runs; it will show the imported namespaces, tables, snapshots, and Parquet files. Save the generated credentials in a private file and load them into your shell:&lt;/p&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nb&quot;&gt;umask &lt;/span&gt;077
docker &lt;span class=&quot;nb&quot;&gt;exec &lt;/span&gt;crowdb-iceberg crowdb-monitor credentials show &lt;span class=&quot;nt&quot;&gt;--format&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;env&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; ./crowdb-iceberg.env
&lt;span class=&quot;nb&quot;&gt;set&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;-a&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;.&lt;/span&gt; ./crowdb-iceberg.env
&lt;span class=&quot;nb&quot;&gt;set&lt;/span&gt; +a
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The loader and DuckDB use &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ICEBERG_URI&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ICEBERG_TOKEN&lt;/code&gt; from this container. Keep the file private.&lt;/p&gt;

&lt;h2 id=&quot;2-load-the-data&quot;&gt;2. Load the data&lt;/h2&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;python3 &lt;span class=&quot;nt&quot;&gt;-m&lt;/span&gt; venv .venv
&lt;span class=&quot;nb&quot;&gt;.&lt;/span&gt; .venv/bin/activate
python &lt;span class=&quot;nt&quot;&gt;-m&lt;/span&gt; pip &lt;span class=&quot;nb&quot;&gt;install &lt;/span&gt;crowdb-tpc-loader
crowdb-tpc-loader load &lt;span class=&quot;nt&quot;&gt;--benchmark&lt;/span&gt; tpch &lt;span class=&quot;nt&quot;&gt;--sf&lt;/span&gt; 1 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--namespace&lt;/span&gt; tpch_demo &lt;span class=&quot;nt&quot;&gt;--report-file&lt;/span&gt; ./tpch-demo.json
crowdb-tpc-loader load &lt;span class=&quot;nt&quot;&gt;--benchmark&lt;/span&gt; tpcds &lt;span class=&quot;nt&quot;&gt;--sf&lt;/span&gt; 1 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--namespace&lt;/span&gt; tpcds_demo &lt;span class=&quot;nt&quot;&gt;--upload-workers&lt;/span&gt; 4 &lt;span class=&quot;nt&quot;&gt;--report-file&lt;/span&gt; ./tpcds-demo.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The &lt;a href=&quot;https://github.com/giancao/crowdb-tpc-loader&quot;&gt;crowdb-tpc-loader&lt;/a&gt; loads each dataset into its own namespace. At SF1, TPC-H creates eight tables and TPC-DS creates 24 tables. Check the completed table counts, generated row totals, and status in your own reports:&lt;/p&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;python - &lt;span class=&quot;o&quot;&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&apos;&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;PY&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&apos;
import json

for name in (&quot;tpch&quot;, &quot;tpcds&quot;):
    with open(f&quot;{name}-demo.json&quot;, encoding=&quot;utf-8&quot;) as file:
        report = json.load(file)
    summary = report[&quot;summary&quot;]
    print(f&quot;{name}: {len(summary[&apos;succeeded&apos;])} tables, {summary[&apos;generated_rows&apos;]:,} rows; status={report[&apos;status&apos;]}&quot;)
    if name == &quot;tpcds&quot;:
        print(f&quot;date_dim: {report[&apos;tables&apos;][&apos;date_dim&apos;][&apos;row_count&apos;]:,} rows&quot;)
&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;PY
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Check that the reports show eight successful TPC-H tables and 24 successful TPC-DS tables, both with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;status=succeeded&lt;/code&gt;. The script also prints the generated row totals and the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;date_dim&lt;/code&gt; row count to compare with the DuckDB query below.&lt;/p&gt;

&lt;p&gt;If an import fails, keep its JSON report and follow the &lt;a href=&quot;https://github.com/giancao/crowdb-tpc-loader/blob/main/docs/RECOVERY.md&quot;&gt;loader recovery guide&lt;/a&gt; before retrying. A new namespace avoids colliding with tables from an earlier run.&lt;/p&gt;

&lt;h3 id=&quot;inspect-the-table-files&quot;&gt;Inspect the table files&lt;/h3&gt;

&lt;p&gt;The development container’s Iceberg tab shows the table tree, snapshot, manifest, and the selected Parquet file. The screenshot below is stopped on the Parquet file page so its row groups, byte layout, and column metadata are visible; the Healthy indicator remains visible in the Console header.&lt;/p&gt;

&lt;figure class=&quot;console-shot&quot;&gt;
  &lt;a href=&quot;/assets/crowdb-iceberg-latest-manifest.png&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot; aria-label=&quot;Open the Iceberg manifest and file screenshot at full size&quot;&gt;&lt;img src=&quot;/assets/crowdb-iceberg-latest-manifest.png&quot; alt=&quot;From a table snapshot and manifest to a Parquet file: byte layout, row groups, and column metadata.&quot; width=&quot;1621&quot; height=&quot;868&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; /&gt;&lt;/a&gt;
  &lt;figcaption&gt;Iceberg file inspection in a local development-container run. The TPC-H lineitem file contains 6,001,215 rows; the view includes the Parquet file, byte layout, row groups, and column metadata. The screenshot uses a run-specific namespace; the commands above use `tpch_demo`. &lt;a href=&quot;/assets/crowdb-iceberg-latest-manifest.png&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;View full-size screenshot ↗&lt;/a&gt;&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;h2 id=&quot;3-query-a-table-with-duckdb&quot;&gt;3. Query a table with DuckDB&lt;/h2&gt;

&lt;p&gt;The empty warehouse selector in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ATTACH &apos;&apos;&lt;/code&gt; is required for this CROWDB profile. Run the DuckDB CLI in the same shell where you loaded the credentials:&lt;/p&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;duckdb &lt;span class=&quot;o&quot;&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SQL&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;
INSTALL iceberg;
LOAD iceberg;
INSTALL httpfs;
LOAD httpfs;
CREATE SECRET crowdb_catalog (TYPE ICEBERG, TOKEN &apos;&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$ICEBERG_TOKEN&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&apos;);
ATTACH &apos;&apos; AS crowdb (TYPE ICEBERG, SECRET crowdb_catalog, ENDPOINT &apos;&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$ICEBERG_URI&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&apos;);
SELECT r_regionkey, r_name FROM crowdb.tpch_demo.region WHERE r_regionkey = 1;
SELECT count(*) AS date_dim_rows FROM crowdb.tpcds_demo.date_dim;
&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SQL
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;For the TPC-H data above, the first query returns &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(1, AMERICA)&lt;/code&gt;. The second count should match &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;date_dim&lt;/code&gt; in your TPC-DS report. Both reads go through CROWDB’s Iceberg catalog and file service.&lt;/p&gt;

&lt;p&gt;The &lt;a href=&quot;https://github.com/giancao/crowdb-tpc-loader/blob/main/docs/TEST_REPORT.md&quot;&gt;recorded SF1 development checks&lt;/a&gt; for loader 0.1.1 committed all 32 tables against a fresh local single-node image. Verification covered remote MD5 comparison, full-row Parquet decoding, Iceberg sample scans, and DuckDB row-count queries for every table. These checks validate the import and read path; they are not timed TPC benchmark results. The &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;latest&lt;/code&gt; image may change, so keep its digest when recording new results.&lt;/p&gt;

&lt;p&gt;When finished, remove the disposable container and credentials file:&lt;/p&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;docker stop &lt;span class=&quot;nt&quot;&gt;--time&lt;/span&gt; 120 crowdb-iceberg
docker &lt;span class=&quot;nb&quot;&gt;rm &lt;/span&gt;crowdb-iceberg
&lt;span class=&quot;nb&quot;&gt;rm&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;-f&lt;/span&gt; ./crowdb-iceberg.env
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
</content></entry><entry><title>Write an Iceberg table with CROWDB and pandas</title><link href="https://giancao.github.io/blog/run-iceberg-in-one-container/"/><id>https://giancao.github.io/blog/run-iceberg-in-one-container/</id><published>2026-09-28T02:30:00+00:00</published><updated>2026-10-02T02:00:00+00:00</updated><summary>Start one container, write six orders to an Iceberg table, and see the pandas result.</summary><content type="html">&lt;p&gt;CROWDB’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;crowdb-iceberg:latest&lt;/code&gt; container runs an Iceberg catalog, table storage, and a local web Console together. You will write six orders to a table, read them back, and total the paid orders by city.&lt;/p&gt;

&lt;p&gt;Use a Linux amd64 host with Docker and Python 3.10–3.12. Use disposable data with this evaluation image, and make sure local port 9092 is free.&lt;/p&gt;

&lt;h2 id=&quot;1-start-the-container&quot;&gt;1. Start the container&lt;/h2&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;docker run &lt;span class=&quot;nt&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;--name&lt;/span&gt; crowdb-iceberg &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;-p&lt;/span&gt; 127.0.0.1:9090:9090 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;-p&lt;/span&gt; 127.0.0.1:9092:9092 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  crowdb/crowdb-iceberg:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;9092&lt;/code&gt; mapping keeps the catalog and file service on your machine. The &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;9090&lt;/code&gt; mapping serves the local Console. Open &lt;a href=&quot;http://127.0.0.1:9090/&quot;&gt;http://127.0.0.1:9090/&lt;/a&gt; in a browser and select &lt;strong&gt;Iceberg&lt;/strong&gt;. The table tree is empty until the script below creates &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pandas_demo.orders&lt;/code&gt;.&lt;/p&gt;

&lt;h2 id=&quot;2-get-credentials-and-install-the-client&quot;&gt;2. Get credentials and install the client&lt;/h2&gt;

&lt;p&gt;The container generates its own connection values. Save them in a private file, then load them into your shell:&lt;/p&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nb&quot;&gt;umask &lt;/span&gt;077
docker &lt;span class=&quot;nb&quot;&gt;exec &lt;/span&gt;crowdb-iceberg crowdb-monitor credentials show &lt;span class=&quot;nt&quot;&gt;--format&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;env&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; ./crowdb-iceberg.env
&lt;span class=&quot;nb&quot;&gt;set&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;-a&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;.&lt;/span&gt; ./crowdb-iceberg.env
&lt;span class=&quot;nb&quot;&gt;set&lt;/span&gt; +a
python3 &lt;span class=&quot;nt&quot;&gt;-m&lt;/span&gt; venv .venv
&lt;span class=&quot;nb&quot;&gt;.&lt;/span&gt; .venv/bin/activate
python &lt;span class=&quot;nt&quot;&gt;-m&lt;/span&gt; pip &lt;span class=&quot;nb&quot;&gt;install&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;pyiceberg[pyarrow]==0.11.1&apos;&lt;/span&gt; pandas
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The file contains &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ICEBERG_URI&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ICEBERG_TOKEN&lt;/code&gt;. Keep it private. The commands below use the values from your container.&lt;/p&gt;

&lt;h2 id=&quot;3-write-six-orders-and-read-them-back&quot;&gt;3. Write six orders and read them back&lt;/h2&gt;

&lt;p&gt;Copy this complete script into &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;orders.py&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;os&lt;/span&gt;

&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pandas&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pd&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pyarrow&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pa&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pyiceberg.catalog&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;load_catalog&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;catalog&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;load_catalog&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;crowdb&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;nb&quot;&gt;type&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;rest&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;uri&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;os&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;environ&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;ICEBERG_URI&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;token&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;os&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;environ&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;ICEBERG_TOKEN&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nc&quot;&gt;DataFrame&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Beijing&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;paid&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;120&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Shanghai&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;paid&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;80&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Beijing&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;cancelled&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;200&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Shanghai&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;paid&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;60&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Shenzhen&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;paid&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;50&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;6&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Beijing&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;paid&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;30&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;columns&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;order_id&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;city&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;status&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;amount_usd&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;arrow_orders&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pa&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Table&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;from_pandas&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;preserve_index&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;False&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;catalog&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;create_namespace_if_not_exists&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;pandas_demo&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;table&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;catalog&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;create_table&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;pandas_demo.orders&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;schema&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;arrow_orders&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;schema&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;table&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;append&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;arrow_orders&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;saved_orders&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;catalog&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;load_table&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;pandas_demo.orders&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;scan&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;().&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;to_pandas&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;saved_orders&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;saved_orders&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;status&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;==&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;paid&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;groupby&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;city&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;as_index&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;False&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;agg&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;orders&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;order_id&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;count&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;revenue_usd&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;amount_usd&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;sort_values&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;revenue_usd&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ascending&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;False&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;nf&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;result&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;to_string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;index&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;False&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Run it once:&lt;/p&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;python orders.py
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Refresh the Console and select &lt;strong&gt;Iceberg → pandas_demo → orders&lt;/strong&gt; to inspect the table, its snapshot, and the Parquet file written by the script. The Console is an inspection surface for this local container; the client and query examples above still use the Iceberg endpoint on &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;9092&lt;/code&gt;.&lt;/p&gt;

&lt;figure class=&quot;console-shot&quot;&gt;
  &lt;a href=&quot;/assets/crowdb-cluster-latest.png&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot; aria-label=&quot;Open the CROWDB cluster screenshot at full size&quot;&gt;&lt;img src=&quot;/assets/crowdb-cluster-latest.png&quot; alt=&quot;CROWDB Console cluster view showing Healthy service status.&quot; width=&quot;1621&quot; height=&quot;868&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; /&gt;&lt;/a&gt;
  &lt;figcaption&gt;The same local container also exposes the Cluster view, where the service health is visible. &lt;a href=&quot;/assets/crowdb-cluster-latest.png&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;View full-size screenshot ↗&lt;/a&gt;&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;figure class=&quot;console-shot&quot;&gt;
  &lt;a href=&quot;/assets/crowdb-iceberg-orders.png&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot; aria-label=&quot;Open a CROWDB Iceberg Console screenshot at full size&quot;&gt;&lt;img src=&quot;/assets/crowdb-iceberg-orders.png&quot; alt=&quot;CROWDB Console showing the real pandas_demo.orders Parquet file from crowdb-iceberg:latest, with Healthy visible.&quot; width=&quot;1621&quot; height=&quot;868&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; /&gt;&lt;/a&gt;
  &lt;figcaption&gt;The Iceberg view follows the real `pandas_demo.orders` table from its snapshot into the manifest and Parquet file. The Console header shows Healthy; the exact generated file name varies by run. &lt;a href=&quot;/assets/crowdb-iceberg-orders.png&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;View full-size screenshot ↗&lt;/a&gt;&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;The fresh Iceberg scan reads the rows saved in the container. One order is cancelled, so the five paid orders produce:&lt;/p&gt;

&lt;div class=&quot;language-text highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;    city  orders  revenue_usd
 Beijing       2          150
Shanghai       2          140
Shenzhen       1           50
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The script creates &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pandas_demo.orders&lt;/code&gt;; running it again with the same container will find that table already exists. To repeat the example, use a new table name or a new disposable container.&lt;/p&gt;

&lt;p&gt;When finished, remove the disposable container and credentials file:&lt;/p&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;docker stop &lt;span class=&quot;nt&quot;&gt;--time&lt;/span&gt; 120 crowdb-iceberg
docker &lt;span class=&quot;nb&quot;&gt;rm &lt;/span&gt;crowdb-iceberg
&lt;span class=&quot;nb&quot;&gt;rm&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;-f&lt;/span&gt; ./crowdb-iceberg.env
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;For a persistent volume or other client operations, see the &lt;a href=&quot;https://crowdb.dev/docs/quickstart/&quot;&gt;quick start&lt;/a&gt; and &lt;a href=&quot;https://crowdb.dev/docs/manual/iceberg/&quot;&gt;Iceberg manual&lt;/a&gt;.&lt;/p&gt;
</content></entry><entry><title>Why we’re building CROWDB</title><link href="https://giancao.github.io/blog/why-we-are-building-crowdb/"/><id>https://giancao.github.io/blog/why-we-are-building-crowdb/</id><published>2026-09-21T02:30:00+00:00</published><updated>2026-10-02T02:00:00+00:00</updated><summary>Why CROWDB owns the path from an Iceberg table to stored data—and what you can test today.</summary><content type="html">&lt;p&gt;When an application reads an Iceberg table, it asks a catalog which files belong to a snapshot, then reads those files. If the catalog, file service, and storage engine are separate systems, even a small change to this path can cross several owners. I’m building CROWDB to make those storage decisions in one system.&lt;/p&gt;

&lt;h2 id=&quot;the-problem-i-want-to-solve&quot;&gt;The problem I want to solve&lt;/h2&gt;

&lt;p&gt;Take a writer that uploads a new table file, then loses its connection before it learns whether the table commit succeeded. Which part knows whether a retry is safe? The catalog knows the snapshot; the file service knows the upload; storage knows where the bytes went. A useful answer needs all three.&lt;/p&gt;

&lt;p&gt;S3 is still useful for objects. An Iceberg table has different rules: files belong to snapshots, and a commit changes which snapshot readers see. I want those rules to be part of CROWDB’s Iceberg path, with the storage work underneath shared where it makes sense.&lt;/p&gt;

&lt;h2 id=&quot;what-crowdb-is&quot;&gt;What CROWDB is&lt;/h2&gt;

&lt;p&gt;CROWDB is a distributed storage platform with separate S3 object and Iceberg table interfaces over a shared storage core.&lt;/p&gt;

&lt;p&gt;An S3 client works with objects. An Iceberg client works with a catalog, tables, snapshots, and files. CROWDB provides the Iceberg catalog and file service together; users do not have to assemble a separate catalog and S3 service for the &lt;a href=&quot;https://giancao.github.io/blog/run-iceberg-in-one-container/&quot;&gt;container example&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Uploading an S3 object does not register an Iceberg table. The two interfaces share storage machinery, but keep their own rules.&lt;/p&gt;

&lt;h2 id=&quot;why-build-the-whole-path&quot;&gt;Why build the whole path?&lt;/h2&gt;

&lt;p&gt;Building a storage core is expensive. CROWDB takes on that work because table commits, file writes, placement, and recovery need clear agreements. For example, after a process dies during a write, the catalog and storage layer must agree on what became durable before a client retries.&lt;/p&gt;

&lt;p&gt;Paxos, write-ahead logs, trees, and erasure coding are established tools. The hard part is making them agree on durability and recovery for the request the client actually made. This design does not prove CROWDB is faster; that needs measurements with a stated workload, hardware, concurrency, and baseline.&lt;/p&gt;

&lt;h2 id=&quot;three-layers-one-system&quot;&gt;Three layers, one system&lt;/h2&gt;

&lt;p&gt;The diagram shows which parts depend on others. It does not show every byte passing through every box.&lt;/p&gt;

&lt;figure class=&quot;architecture-figure&quot; aria-label=&quot;CROWDB architecture: access models, chunk layer, and reusable KV&quot;&gt;
  &lt;div class=&quot;figure-header&quot;&gt;&lt;span&gt;FIG. 01 / THREE LAYERS&lt;/span&gt;&lt;span class=&quot;figure-legend&quot;&gt;&lt;span&gt;&lt;i&gt;&lt;/i&gt;implemented&lt;/span&gt;&lt;span&gt;&lt;i class=&quot;dashed&quot;&gt;&lt;/i&gt;planned&lt;/span&gt;&lt;/span&gt;&lt;/div&gt;
  &lt;div class=&quot;access-cards&quot;&gt;
    &lt;div class=&quot;access-card&quot;&gt;&lt;b&gt;S3&lt;/b&gt;&lt;span&gt;HTTP objects&lt;/span&gt;&lt;span class=&quot;tiny-status&quot;&gt;IMPLEMENTED&lt;/span&gt;&lt;/div&gt;
    &lt;div class=&quot;access-card&quot;&gt;&lt;b&gt;Iceberg&lt;/b&gt;&lt;span&gt;Catalog + FileIO&lt;/span&gt;&lt;span class=&quot;tiny-status&quot;&gt;IMPLEMENTED&lt;/span&gt;&lt;/div&gt;
    &lt;div class=&quot;access-card planned&quot;&gt;&lt;b&gt;Dataset&lt;/b&gt;&lt;span&gt;HTTP + native client&lt;/span&gt;&lt;span class=&quot;tiny-status&quot;&gt;IN DESIGN&lt;/span&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class=&quot;figure-connector&quot; aria-hidden=&quot;true&quot;&gt;↓&lt;/div&gt;
  &lt;div class=&quot;layer-box dark&quot;&gt;&lt;div class=&quot;layer-title&quot;&gt;&lt;span&gt;02&lt;/span&gt;Chunk&lt;/div&gt;&lt;div class=&quot;layer-main&quot;&gt;&lt;b&gt;One protected storage layer&lt;/b&gt;&lt;p&gt;Chunk Stream · Chunk-KV&lt;br /&gt;Placement, bounded streaming, protection and repair&lt;/p&gt;&lt;div class=&quot;path-line&quot;&gt;chunk client → Chunk I/O → ChunkDB → DiskIO → DiskDB&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;
  &lt;div class=&quot;figure-connector&quot; aria-hidden=&quot;true&quot;&gt;↕&lt;/div&gt;
  &lt;div class=&quot;layer-box&quot;&gt;&lt;div class=&quot;layer-title&quot;&gt;&lt;span&gt;01&lt;/span&gt;Reusable KV&lt;/div&gt;&lt;div class=&quot;layer-main&quot;&gt;&lt;b&gt;Replicated metadata and durable state&lt;/b&gt;&lt;p&gt;crowdb-kv · Multi-Paxos · WAL · crowdb-tree · RPC&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;
  &lt;div class=&quot;planned-path&quot;&gt;&lt;strong&gt;PLANNED PATH&lt;/strong&gt; &amp;nbsp; DiskIO buffers → RDMA / GDS → GPU memory. This is a design direction, not a capability of the preview.&lt;/div&gt;
  &lt;figcaption&gt;The KV layer supplies distributed state; the diagram does not imply that every payload is stored in KV. S3 and Iceberg use the shared core without translating through one another.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;h3 id=&quot;access-models-keep-their-meaning&quot;&gt;Access models keep their meaning&lt;/h3&gt;

&lt;p&gt;The Access layer serves S3 object requests and Iceberg catalog and file requests. Each interface keeps its own rules for what a write means.&lt;/p&gt;

&lt;h3 id=&quot;chunks-own-the-storage-work&quot;&gt;Chunks own the storage work&lt;/h3&gt;

&lt;p&gt;The Chunk layer places and protects stored data. Both access models can use it without each building separate placement and recovery code.&lt;/p&gt;

&lt;h3 id=&quot;distributed-state-is-a-reusable-foundation&quot;&gt;Distributed state is a reusable foundation&lt;/h3&gt;

&lt;p&gt;Underneath, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;crowdb-kv&lt;/code&gt; replicates the small pieces of state those layers need to agree on. It is a reusable part of the system, not the model an Iceberg or S3 client sees.&lt;/p&gt;

&lt;h2 id=&quot;where-the-project-stands&quot;&gt;Where the project stands&lt;/h2&gt;

&lt;div class=&quot;callout&quot;&gt;
&lt;strong&gt;Development status · updated October 2, 2026&lt;/strong&gt;
&lt;p&gt;S3 and native Iceberg have working implementations. Dataset and direct GPU delivery are still in design. The single-node container is for disposable data: it has no host fault tolerance, production support, or supported on-disk upgrade path.&lt;/p&gt;
&lt;/div&gt;

&lt;p&gt;To check the current Iceberg path yourself, start the disposable container on a Linux amd64 host with Docker and a free local port 9092:&lt;/p&gt;

&lt;div class=&quot;language-sh highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;docker run &lt;span class=&quot;nt&quot;&gt;-d&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;--name&lt;/span&gt; crowdb-iceberg &lt;span class=&quot;nt&quot;&gt;-p&lt;/span&gt; 127.0.0.1:9092:9092 crowdb/crowdb-iceberg:latest
docker &lt;span class=&quot;nb&quot;&gt;exec &lt;/span&gt;crowdb-iceberg crowdb-monitor readiness &lt;span class=&quot;o&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;echo &lt;/span&gt;ready
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Rerun the readiness command until it prints &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ready&lt;/code&gt;, then start at step 2 of the &lt;a href=&quot;https://giancao.github.io/blog/run-iceberg-in-one-container/&quot;&gt;six-order walkthrough&lt;/a&gt; to write a table and read it back in pandas. The example’s five paid orders produce city totals of 150, 140, and 50 USD. The guide includes the complete script, expected output, and cleanup.&lt;/p&gt;

&lt;div class=&quot;source-note&quot;&gt;Technical references: &lt;a href=&quot;https://github.com/giancao/crowdb&quot;&gt;project README and status&lt;/a&gt;, &lt;a href=&quot;https://github.com/giancao/crowdb/tree/main/doc/design&quot;&gt;design documents&lt;/a&gt;, and the &lt;a href=&quot;https://crowdb.dev/docs/quickstart/&quot;&gt;Iceberg quick start&lt;/a&gt;. First published September 21, 2026; updated October 2, 2026.&lt;/div&gt;
</content></entry>
</feed>