Refactor header gen (#28)

* Update headergen

* Add changeset

* Fix linting issue

* Update docs

* Updatmp dir issue

* Update outputs for header gen

* Update docs with file directory changes

* Clarify primitive type table in c-proto-wire-mapping.md

- Define W32/W64 as the --word-size flag (target platform ABI)
- Note that plain char signedness is delegated to libclang at parse time
- Clarify that fixed-size arrays use Vec for ergonomics but the decoder
  reads exactly N elements baked into the generated struct

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Handle failed disk writes

* Update docs

* Cargo fmt

* Update docs

* Fail loudly

* Revert to c based offsets

* Update reference doc

* Update tests

* Update to major

* Update docs

* Use i32:Max

* Update dockerfile with stale reference

* Add roundtrip tests

* Add example headers

* Update readme

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: Codykilpatrick

.changeset/header-gen-align-outputs.md

@@ -0,0 +1,17 @@
+---
+"@spear-ai/webway": major
+---
+
+`header-gen` outputs now match `wsdl-gen` in contract and wire format (MOD-158).
+
+**Breaking changes:**
+- Output file renamed `structs.rs` → `messages.rs`
+- `mappers.rs` and `--out-mapping` CLI flag removed
+- `decode_raw` / `encode_raw` now use **C struct ABI layout** (libclang field offsets, includes alignment padding) — the only correct interpretation for binary messages produced by C programs
+
+**New capabilities:**
+- Generated structs carry `#[derive(prost::Message, Serialize, Deserialize)]`
+- Four codec methods on every struct: `decode_raw`, `decode` (with endianness prefix), `encode_raw`, `encoded_size`
+- C enums are now parsed and emitted as Rust `#[derive(prost::Enumeration)]` enums and proto3 `enum` blocks, following MOD-140 conventions (`{EnumName}Unspecified = 0`, sequential variants with `// c: N` comments, `{EnumName}Garbage = 2147483647`)
+- Proto output restructured to per-header shards at `types/headers/{stem}.proto` with `package spear.v1.{stem};` and cross-header imports (mirrors PR #24 for `wsdl-gen`)
+- `--out-proto` is now the proto tree root; shards are written under `types/headers/` within it

.changeset/header-gen-inline-nested-types.md

@@ -0,0 +1,9 @@
+---
+"@spear-ai/webway": patch
+---
+
+`header-gen` now registers inline nested struct and enum definitions (MOD-164).
+
+Previously, types defined inline inside a parent struct (e.g. `struct Moon { ... } moon;` or `enum FlightPhase { ... } phase;`) were silently omitted from the registry, causing the emitter to generate decode calls referencing types that didn't exist. Now these nested types are discovered when their parent struct's fields are traversed and are registered with the same input-directory filter applied to top-level declarations.
+
+All three inline-nesting patterns are supported: single nested struct field, fixed-size array of nested structs, and nested enum field.

.cursor/rules/technical-decisions-mod-140.mdc

@@ -30,10 +30,10 @@ Canonical source: [MOD-140 on Linear](https://linear.app/spear-ai/issue/MOD-140/
 
 | | Template |
 | -- | -- |
-| Type package (xsd) | `spear.v1.{schema_filename}` |
-| Type package (headers) | `spear.v1.{header_filename}` |
-| Type file (xsd) | `proto/types/xsd/{schema_filename}.proto` |
-| Type file (headers) | `proto/types/headers/{header_filename}.proto` |
+| Type package (xsd) | `spear.v1.{schema_relative_stem}` (e.g. `spear.v1.net.types`) |
+| Type package (headers) | `spear.v1.{header_relative_stem}` (e.g. `spear.v1.net.messages`) |
+| Type file (xsd) | `proto/types/xsd/{schema_relative_stem}.proto` (e.g. `proto/types/xsd/net/types.proto`) |
+| Type file (headers) | `proto/types/headers/{header_relative_stem}.proto` (e.g. `proto/types/headers/net/messages.proto`) |
 | Message package | `spear.message.v1.{header_type}.{payload_type}` |
 | Message file | `proto/messages/{header_type}/{payload_type}.proto` |
 | Schema registry subject | `spear.message.v1.{header_type}.{payload_type}` |
@@ -53,7 +53,7 @@ Two directories under `proto/`:
 
 **Package naming**
 
-- Types: `spear.v1.{schema_filename}` — derived mechanically from source file name, no manual config.
+- Types: `spear.v1.{relative_stem}` — derived mechanically from the source file's path relative to the input directory (e.g. `net/messages.h` → `spear.v1.net.messages`). Directory structure is preserved so same-named files in different subdirectories produce distinct packages. No manual config.
 - Messages: `spear.message.v1.{header_type}.{payload_type}` — transport (SLEMR/DCM) is excluded from package names. `spear` is used as the root over effort-specific names for long-term stability.
 
 **Message naming**
@@ -158,7 +158,7 @@ C code comparing against XSD integer values can migrate using the `// xsd: N` co
 
 **Versioning**
 
-Version lives in the package path only — `spear.v1.{schema_filename}` for types, `spear.message.v1.{header_type}.{payload_type}` for messages. No version in individual message or type names. A version bump happens at the `messages/` layer when consumers would need to change their code. Non-breaking changes (adding an optional field) do not require a version bump.
+Version lives in the package path only — `spear.v1.{relative_stem}` for types, `spear.message.v1.{header_type}.{payload_type}` for messages. No version in individual message or type names. A version bump happens at the `messages/` layer when consumers would need to change their code. Non-breaking changes (adding an optional field) do not require a version bump.
 
 **Proto syntax**
 

CLAUDE.md

@@ -32,8 +32,8 @@ cargo test -p spear-lib
 # XSD → proto + Rust
 cargo run -p wsdl-gen -- --input schemas/synthetic --out-proto generated/proto --out-rust generated/rust
 
-# C headers → Rust structs + proto + mapping
-cargo run -p header-gen -- --input headers/ --include /usr/include --endian little --word-size 32 --define LINUX --out-rust generated/rust --out-proto generated/proto --out-mapping generated/mapping
+# C headers → Rust structs + proto (per-header shards under types/headers/)
+cargo run -p header-gen -- --input examples/headers/ --include /usr/include --endian little --word-size 32 --define LINUX --out-rust generated/rust --out-proto generated/proto
 ```
 
 ## Architecture
@@ -44,7 +44,7 @@ Spear is a **Data Normalization Gateway** — it decodes legacy binary messages
 
 ```text
 .xsd files  →  wsdl-gen   →  .proto + .rs (decode/encode methods)
-.h files    →  header-gen  →  Rust structs + .proto + mapping functions
+.h files    →  header-gen  →  messages.rs + types/headers/*.proto (per-header shards)
                                   ↓
                Raw binary  →  spear-gateway  →  decoded output
                                   ↓  (Phase 3, planned)
@@ -56,7 +56,7 @@ Spear is a **Data Normalization Gateway** — it decodes legacy binary messages
 | Crate | Purpose |
 |---|---|
 | `wsdl-gen` | XSD → proto3 + Rust with `decode_raw`/`encode_raw`/`encoded_size` |
-| `header-gen` | C headers (via libclang) → Rust structs + proto + mapping fns |
+| `header-gen` | C headers (via libclang) → `messages.rs` + proto3 shards with `decode_raw`/`encode_raw`/`encoded_size` |
 | `spear-lib` | Runtime library: WSDL parser, ProtoEnvelope, Redpanda publisher |
 | `spear-gateway` | Decode pipeline: raw bytes → generated types → printed output |
 

Cargo.lock

@@ -14,7 +14,7 @@
 # Inside the container:
 #   wsdl-gen --input /workspace/xsds --out-proto /workspace/proto --out-rust /workspace/rust
 #   header-gen --input /workspace/headers --endian little --word-size 32 \
-#              --out-rust /workspace/rust --out-proto /workspace/proto --out-mapping /workspace/mapping
+#              --out-rust /workspace/rust --out-proto /workspace/proto
 #   cp /workspace/types.rs /spear/crates/spear-gateway/src/types.rs
 #   # uncomment include!("types.rs") in main.rs, add decode call
 #   cargo build --offline --release -p spear-gateway

Dockerfile

@@ -25,9 +25,10 @@ npm install
 │                           └──► types.rs                     │
 │                           (decode / decode_raw / encoded_size)│
 │                                                             │
-│   .h files    ──► header-gen ──► structs.rs  (decode())     │
-│                           ├──► messages.proto               │
-│                           ├──► mappers.rs                   │
+│   .h files    ──► header-gen ──► messages.rs                │
+│                           │    (decode_raw/encode_raw/       │
+│                           │     encoded_size)                │
+│                           ├──► types/headers/{stem}.proto   │
 │                           └──► review_report.txt            │
 └─────────────────────────────────────────────────────────────┘
 
@@ -61,7 +62,7 @@ npm install
 | Crate | Type | Purpose |
 |---|---|---|
 | `wsdl-gen` | binary | Code generator: XSD → `.proto` + `.rs` with `decode_raw`/`encoded_size` |
-| `header-gen` | binary | Code generator: C headers → Rust structs + `.proto` + mapping functions |
+| `header-gen` | binary | Code generator: C headers → `messages.rs` + per-header `.proto` shards with `decode_raw`/`encode_raw`/`encoded_size` |
 | `spear-lib` | library | Runtime: WSDL parser, `ProtoEnvelope`, Redpanda publisher |
 | `spear-gateway` | binary | Decode pipeline: raw binary bytes → generated types → printed output |
 
@@ -106,11 +107,12 @@ XSD → proto3/Rust mapping rules.
 
 ## header-gen: C header → code generation
 
-Takes a directory of `.h` files and emits three files per struct:
+Takes a directory of `.h` files and emits:
 
-- `--out-rust` — Rust structs with `decode(bytes: &[u8])` (offset-based, configurable endianness) + `review_report.txt` for anything requiring manual review (bitfields, unions, unresolved types)
-- `--out-proto` — proto3 message definitions
-- `--out-mapping` — explicit `map_*()` functions from each Rust struct to its proto message
+- `--out-rust` — `messages.rs` with `prost::Message` + serde derives, and `decode_raw`/`decode`/`encode_raw`/`encoded_size` methods using the C struct binary layout (field positions and struct size come from libclang — matches the in-memory ABI layout produced by the C compiler, including alignment padding). Also writes `review_report.txt` for constructs requiring manual review (bitfields, unions, unresolved types).
+- `--out-proto` — per-header proto3 shards at `types/headers/{stem}.proto`, one per input `.h` file, with `package spear.v1.{stem};` and cross-header imports.
+
+See [docs/c-proto-wire-mapping.md](docs/c-proto-wire-mapping.md) for the full type mapping table and wire format details.
 
 **From the spear-dev container** (recommended — no system libclang install required):
 
@@ -123,29 +125,27 @@ podman exec -it spear-dev header-gen \
   --word-size  32 \
   --define     LINUX \
   --out-rust   /workspace/generated/rust \
-  --out-proto  /workspace/generated/proto \
-  --out-mapping /workspace/generated/mapping
+  --out-proto  /workspace/generated/proto
 ```
 
 **From source** (requires Rust + libclang):
 
 ```bash
 cargo run -p header-gen -- \
-  --input      headers/ \
+  --input      examples/headers/ \
   --include    /usr/include \
   --endian     little \
   --word-size  32 \
   --define     LINUX \
   --out-rust   generated/rust \
-  --out-proto  generated/proto \
-  --out-mapping generated/mapping
+  --out-proto  generated/proto
 ```
 
-`--word-size` controls how `long`/`unsigned long` are mapped:
+`--word-size` is passed to clang as `-m32` (or left as native for 64-bit) so clang computes the correct ABI layout for the target. It also controls how `long`/`unsigned long` are mapped in the generated types:
 - `32` → `i32`/`u32` (LP32/ILP32 ABI)
 - `64` → `i64`/`u64` (LP64 ABI)
 
-`--endian` controls the decode method emitted (`from_le_bytes` vs `from_be_bytes`).
+`--endian` is passed to clang as `-mbig-endian` when set to `big`, so clang computes the correct target layout. It also controls the byte order used by `decode_raw`/`encode_raw` (passed as `same_endianness` to the `_codec` helpers).
 
 `--include PATH` (repeatable) adds an extra clang include search path. Use this for
 system headers or cross-compilation sysroots that your `--input` headers `#include`.
@@ -230,8 +230,7 @@ header-gen \
   --word-size  32 \
   --define     LINUX \
   --out-rust   /workspace/generated/rust \
-  --out-proto  /workspace/generated/proto \
-  --out-mapping /workspace/generated/mapping
+  --out-proto  /workspace/generated/proto
 ```
 
 ### 4. Plug in generated types and rebuild

README.md

@@ -13,6 +13,8 @@ clap = { workspace = true }
 anyhow = { workspace = true }
 clang = { workspace = true }
 clang-sys = { version = "1" }
+prost = { workspace = true }
+serde = { workspace = true }
 
 [dev-dependencies]
 tempfile = { workspace = true }

crates/header-gen/Cargo.toml

@@ -1,4 +1,3 @@
-pub mod mapping;
 pub mod proto;
 pub mod rust_structs;