README.md

Apache Iceberg Official Native Rust Implementation

This crate contains the official Native Rust implementation of Apache Iceberg.

See the API documentation for examples and the full API.

Usage

use futures::TryStreamExt;
use iceberg::io::{FileIO, FileIOBuilder};
use iceberg::{Catalog, Result, TableIdent};
use iceberg_catalog_memory::MemoryCatalog;

#[tokio::main]
async fn main() -> Result<()> {
    // Build your file IO.
    let file_io = FileIOBuilder::new("memory").build()?;
    // Connect to a catalog.
    let catalog = MemoryCatalog::new(file_io, None);
    // Load table from catalog.
    let table = catalog
        .load_table(&TableIdent::from_strs(["hello", "world"])?)
        .await?;
    // Build table scan.
    let stream = table
        .scan()
        .select(["name", "id"])
        .build()?
        .to_arrow()
        .await?;

    // Consume this stream like arrow record batch stream.
    let _data: Vec<_> = stream.try_collect().await?;
    Ok(())
}

IO Support

Iceberg Rust provides various storage backends through feature flags. Here are the currently supported storage backends:

Storage Backend	Feature Flag	Status	Description
Memory	storage-memory	✅ Stable	In-memory storage for testing and development
Local Filesystem	storage-fs	✅ Stable	Local filesystem storage
Amazon S3	storage-s3	✅ Stable	Amazon S3 storage
Google Cloud Storage	storage-gcs	✅ Stable	Google Cloud Storage
Alibaba Cloud OSS	storage-oss	🧪 Experimental	Alibaba Cloud Object Storage Service

You can enable all stable storage backends at once using the storage-all feature flag.

Note that storage-oss is currently experimental and not included in storage-all.

Example usage in Cargo.toml:

[dependencies]
iceberg = { version = "x.y.z", features = ["storage-s3", "storage-fs"] }