Repository navigation
feat(zero-rust): Rust client - #130
Merged
Merged
Conversation
…d CI Adds rust/, a Cargo workspace with two crates: - zero-client, the client, starting with the JSON layer: JS property order, JSON.parse and JSON.stringify, Number#toString and Number(string), UTF-8 and UTF-16 string orders, and xxHash32 with h64 and h128. - zero-conformance, which replays packages/zero-conformance's vectors and seeded fuzz corpora and reports, per file, the cases that match JS, the ones that match only under a documented exception, and the ones that fail. Files with no suite yet are listed as not implemented. Replayed so far: string-compare, hash, number-format, number-parse, json-parse, json-stringify-strings, sweep-json, harden-json, and the generated harden-json-fuzz corpus. All 23,757 cases match JS: 2,251 only under the lone-surrogate exception and one under the nesting limit, both as in the Swift client (rust/DESIGN.md). The Rust workflow checks the vectors are current, then runs rustfmt, clippy, the tests, and the conformance report on rust-client. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
compareValues and valuesEqual, toPrimaryKeyString and the aggregate row keys, LIKE and ILIKE on UTF-16 code units as JS's non-Unicode RegExp matches them, and parseTTL, normalizeTTL, clampTTL, and compareTTL. compare-values, primary-key-string, aggregate-row-key, like, and ttl replay with all 655 cases matching JS exactly. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Sets up the Rust client's release the way the Swift client's works: - packages/zero-rust carries the crates' version and changelog for Changesets; it isn't published to npm. - Swift Release's Version Packages PR now versions both clients: it also copies zero-rust's version into rust/Cargo.toml and Cargo.lock (rust/scripts/set-version.sh). A second Changesets job would race it for the same branch, so the PR is now titled "chore: version packages". - rust-release.yml, on main, releases a version that isn't out yet after approval in the rust-release environment: a subtree split of rust/ to wevm/zero-rust with a tag and a GitHub release from the changelog, and cargo publish of zero-client, with trusted publishing once the crate exists. 0.0.0 is never released. - rust/.github/sts/mirror.sts.yaml authorizes the mirror token. - rust.yml adds rustdoc and cargo package checks, and an API-break job (cargo semver-checks) that starts once zero-client is on crates.io. Also adds rust/PARITY.md, the crate README, and a release section in rust/DESIGN.md. The conformance test skips in the mirror, which has no vectors; CI's conformance step still fails without them. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
…mapping The AST and its JSON form (astSchema), strictASTSchema's depth limit and unknown-key rejection, normalizeAST with V8's Array#sort (the comparators are inconsistent, so the sort's exact comparisons decide the hash), hashOfAST, hashOfQueryInternals, and hashOfNameAndArgs, the view Format, and the client-to-server NameMapper over a schema's tables and columns. ast, ast-features, query-internals-hash, name-args-hash, sweep-name-args-hash, and the generated ast-fuzz corpus replay with all 2,288 cases matching JS exactly. ast-features maps names with the JS test schema as JS wrote it into ivm.json, and compares rejections by verdict, not by message. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
…tocols header Upstream and Downstream with every message body: initConnection and the Sec-WebSocket-Protocol header (base64 and decodeURIComponent as atob and the browser do them), query patches, pokes and row patches, pushes, mutation results, error bodies with their kinds, origins, and reasons, the client schema and its normalization, and the inspect messages, checked and stripped by a small interpreter of valita schemas. Binary poke chunks are joined as bytes before strict UTF-8 decoding. PROTOCOL_VERSION is 57, inside meta.json's window. All replayed vectors match JS: protocol-upstream, protocol-downstream, sweep-downstream, protocol-poke-chunks, protocol-sec-protocols, protocol-client-schema, meta, and the generated protocol fuzz, harden-downstream-text, and harden-poke-bytes corpora, 28,139 cases. Loosening one field's decoding fails 26 of them, so the suites do catch a wrong decoder. Also fixes rust/PARITY.md, whose ported table had missed the AST rows. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
AnyQuery and ExpressionBuilder, ported from QueryImpl and expression.ts by way of Swift's AnyQuery, and schema relationships. sweep-query-hash replays all 347 programs, matching JS's AST and hash. This finishes M1. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
The manifest said 1.85, but the library uses let-chains, stable since 1.88; it fails to build on 1.85. The higher MSRV also enables a clippy lint, so xxhash32 uses as_chunks. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
The connect URL (createConnectionURL) and the client schema (clientSchemaFrom), and tests/walking_skeleton.rs: it connects with the initConnection message in Sec-WebSocket-Protocol, asks for issues.open, decodes the pokes, text and binary, and checks the fixture's rows. CI runs it in a new end-to-end job with Postgres and zero-cache, and checks the build on the minimum Rust version. This finishes M1.5. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
The rust-release environment, like swift-release, limits the release jobs to main and has no required reviewers. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Nodes, changes, lazy streams, constraints, row comparison, the operator and source traits, pipeline error capture, and row predicates, ported from packages/zql/src/ivm by way of Swift's IVM. The start of M2. Also records tmm's transport decision: a Transport trait in the core and an opt-in tokio-tungstenite reference transport. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
…cided Records the dependency review: tokio-tungstenite without default features, TLS as its own feature so no crypto provider is pulled in by default, and rusqlite not required until tmm decides how storage is opted into. Also records the crate name, zero-client. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
MemorySource and its connections, ported from memory-source.ts by way of Swift: sorted indexes made on demand, constrained and reverse scans from a bound, the overlay that shows a push in flight to the fetches it triggers, split edits, multi-constraint fetches, and filterPush. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
JSON objects and schemas use map::OrderedMap, an insertion-ordered map in the crate: a linear scan for small maps, such as rows, and a hash index of positions above eight keys. Object's iterators are the crate's own types, so no indexmap type is in the public API. CI now fails when the default build of zero-client has a normal or build dependency. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
The storage design for M5, after zero-swift's: the core keeps the mirror, the decisions, and the lifecycle; a backend trait loads at open and applies an ordered write stream; memory is in the core and is the fallback; SQLite comes behind sqlite and sqlite-bundled features, with writes on a std thread. Whether the default build is memory-only is still open. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
tmm's decision. The README and the crate docs say a default build persists nothing across restarts, and the quick start adds the sqlite feature. The storage options are named as in Swift: memory, SQLite, and a custom store. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
tmm makes the first publish of zero-client himself, with his own cargo login. Until the crate exists the release run skips the crates.io job with a notice, still pushing the mirror and its release; afterwards it publishes with trusted publishing only. The CARGO_REGISTRY_TOKEN fallback is gone. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Ports Filter, FilterStart, FilterEnd, Exists, FanOut, FanIn, UnionFanOut, UnionFanIn, Join, TakeGate, JoinIndexTap, and the join index and overlay helpers from the Swift client. Not yet exercised by conformance. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
…perators Ports FlippedJoin, Take, Skip, Cap, LiftField, Aggregate, GroupBy, GroupProjection, and DeferredInput from the Swift client. Not yet exercised by conformance. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Ports ArrayView and view change application, the pipeline builder in compute mode with correlated predicate pushdown, and Take and Cap's early-return hydration error. The conformance runner now replays ivm, ivm-features, bughunt-ivm, and the ivm, sweep-ivm and harden-ivm corpora: 7,321 of 7,321 cases, 3 under the new unwritable-path exception. Adds a nightly differential run on fresh seeds. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Formats zero-client's Cargo.toml, which Public JS's oxfmt check flagged on the pull request. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
zero_client::typed wraps AnyQuery with row structs, typed column and relationship constants, and result types read from views. zerogen gets a Rust target and a zero-rust CLI (packages/zero-rust-cli, private for now) that share schema loading with zero-swift. rust/codegen-tests holds the generated Rust for the zql test schema, the codegen fixture, zbugs, and every sweep schema, compiled with clippy's warnings denied, and tests that typed queries build the untyped queries' ASTs and that their views read as generated rows. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
…e CLIs syncpack failed on zero-rust-cli's typescript 6.0.3. Like zero-swift and zerogen, it reads schemas with the TypeScript compiler API, which TypeScript 7 doesn't have. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
… version As zero-swift does for Swift, the CLI's npm package now carries the Rust client's version and changelog: packages/zero-rust-cli is the zero-rust package, and the version carrier packages/zero-rust is gone. It is bundled with esbuild like zero-swift, with a packed-package smoke test in CI. rust-release.yml reads the version and changelog from it and publishes it to npm with trusted publishing once the package exists; the first publish is made by hand, as for crates.io. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Port the JS client engine to Rust, after the Swift client's Engine: the connection manager, query manager, poke handler with rebase and replay, mutation tracker, query context and views, CRUD transactions with optimistic aggregates, auth, recovery, and the client and client-group lifecycle, on a thread-local memory database. The socket is behind SocketFactory, and zero_client::testing has a manual scheduler and mock sockets. The conformance runner replays the client scenarios: client 183/183, client-harden-connection 62/62, client-harden-mutations 25/25, harden-queries 41/41, bughunt-client 12/12. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Add zero_client::tokio_transport behind the tokio-transport feature: a Scheduler on a tokio LocalSet and a SocketFactory on tokio-tungstenite, with no tokio or tungstenite type in the public API. The tls feature adds wss:// with rustls on ring and webpki roots. The default build still has no dependencies. tests/tokio_transport.rs checks frames, closes, and timers against a local WebSocket server; tests/client_e2e.rs runs the whole client against zero-cache, as Swift's ClientE2ETests and AggregateE2ETests do. CI lints and tests with every feature and runs the client end-to-end tests. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Generate, per table, <Row>Update (the primary key, an Option per other
column, and a ColumnUpdate for one that can be null) and <Row>Key, and a
mutate module with a function per table returning a TypedTableMutator:
mutate::issue(tx).update(&IssueUpdate { closed: Some(true),
..IssueUpdate::new(id) }), as Swift's tx.mutate.<table>. The names are
claimed after the schema's own types, so existing generated names don't
change.
Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c
Co-authored-by: tmm <tmm@tmm.dev>
The client suite replays generated/client-aggregates-fuzz, the seeded aggregate scenarios of client/bughunt.test.ts, which the runner skipped without listing: it now lists every generated corpus no suite replays as not implemented, as it does checked-in files. Rust Nightly generates them on fresh seeds, as Swift Nightly does, and replays them with the IVM corpora. 150/150 on the PR seed; 3,000/3,000 on four fresh seeds locally. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
Put the client's database behind public StorageBackend and StoredDatabase traits, as zero-swift's ReplicaDatabase: the core keeps the client groups and clients, makes every decision, and writes an ordered StorageOp stream. ZeroOptions gains storage (Memory, the default; Sqlite behind the sqlite feature; Custom) and on_storage_failure. A store that can't open falls back to memory, one that doesn't decode is dropped as lost client state, and the database GC runs five minutes in and every 12 hours. Zero gains delete and flush_storage. SQLite (rusqlite 0.40, the newest on Rust 1.88; sqlite-bundled compiles it in) uses zero-swift's file layout, a writer thread per open database, WAL, and one transaction per write. The client scenarios run again on SQLite, each on a directory of its own: 473/473 match JS. tests/sqlite_storage.rs checks the file itself, and client_e2e relaunches on SQLite against zero-cache. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c Co-authored-by: tmm <tmm@tmm.dev>
tmm
added a commit
that referenced
this pull request
Nov 11, 2026
…the apps' lockfiles (#145) changesets/action runs version-script through @actions/exec without a shell, so the && chain was passed as arguments and the Version PR job failed on every push to main since #130. The script now runs under sh -c. rust/scripts/set-version.sh also bumps zero-client in the zbugs apps' lockfiles so their --locked builds keep working after a release. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Rust Zero client,
zero-client, through milestone M5. The goal is parity with the JS client, proven by conformance against the JS vectors inpackages/zero-conformance. The milestones and design are inrust/DESIGN.md, and what's ported and every documented difference is inrust/PARITY.md. The zbugs apps (M6) and hardening (M7) follow in their own PRs.What's in it
rust/workspace, and a conformance runner that reports every vector file and generated corpus. A file it doesn't replay is listed as not implemented.LIKE, TTL, the AST and its normalization, query hashes, the query builder, name mapping, protocol messages, and binary poke chunks.zero-cache. CI runs it end to end against Postgres.zero-rust codegen(npmzero-rust,packages/zero-rust-cli, on the sharedpackages/zerogen) generates row structs, the schema, typed queries, and typed CRUD for mutators (mutate::issue(tx).update(…)).tokio-transport,tlswith rustls onring).StorageBackendandStoredDatabasetraits;ZeroStorage::Memory(the default, which keeps nothing across restarts),Sqlite(sqliteorsqlite-bundled, zero-swift's file layout), andCustom; memory fallback, corrupt-store recovery, and the database GC;Zero::delete.Conformance: 63,453 of 63,453 cases replayed, 0 failures.
u32index type can't hold (each checked against JS's error message).(sqlite)row:client,client-harden-connection,client-harden-mutations,harden-queries,bughunt-client, and the seededclient-aggregates-fuzz.End to end (CI, against Postgres and
zero-cachefrom this repo): the walking skeleton, and the whole client on the tokio transport: sync, optimistic mutations the server confirms, a relaunch on memory and on SQLite, two clients of one group, and synced aggregates.Dependencies and toolchain
zero-client's default build has no third-party dependencies; CI fails if one appears outside an opt-in feature. tokio, tokio-tungstenite, rustls and rusqlite are opt-in.Outside
rust/rust.yml: CI on every PR.rust-nightly.yml: fresh IVM and client-scenario seeds every night; it starts once this merges, since GitHub runs scheduled workflows only from the default branch.rust-release.yml,packages/zero-rust-cli(npmzero-rust, also the Changesets version carrier),packages/zerogen(the Rust target), and the changeset docs.swift-release.yml: its Version Packages PR now versions both clients and is retitledchore: version packages.packages/zero-conformance/src/e2e/.Merging releases nothing. The crate and CLI are at 0.0.0, which
rust-release.ymlnever publishes, and there's nozero-rustchangeset. The first crates.io and npm publishes are tmm's, by hand. Seedingwevm/zero-rust's main after the merge waits for his go-ahead.Amp thread: https://ampcode.com/threads/T-01a1c6b9-fc99-73a1-bf70-e71fd484042c