neuIP-5: Subset Schema Definition and Identification using JSON Schema and JCS

[neatonk]

Overview

This neuIP proposes a standard for defining, identifying, validating, and deriving subset schemas for Neume. The standard uses JSON Schema and the JSON Canonicalization Scheme (JCS). The aim is to improve data partitioning, dataset reconciliation, and execution orchestration.

Motivation

The proposal intends to address Neume's current lack of a deterministic methodology for identifying subsets and their schemas. It proposes deterministic Schema Identifiers, created through JSON Schema-based definitions and JCS.

Specification

Schema

• Schema Definition: Schemas should be defined as a JavaScript object that adheres to JSON Schema.
• Schema Identification: Each schema should have a unique identifier determined by generating a BLAKE3 hash of the canonicalized schema document.
• Schema Validation: Candidate data items should be validated against the schema to determine set membership using the ajv and ajv-formats packages.
• Schema Derivation: Schemas can be combined with a constraint schema to produce a more specific subset of data.

Strategy

• Strategy Definition: A strategy should be a JavaScript or TypeScript function that receives and generates data conforming to respective input and output schemas.
• Strategy Parameterization: Strategies can be parameterized to operate on specific subsets of the schema. Parameters should be included in the strategy definition.

Additional Requirements

• Data Provenance: Data items must include sufficient information to validate the source of the data. For blockchain-derived data, this should include the CAIP-2 chain id, contract address, and token id.

Rationale

This proposal leverages JSON Schema for schema definition and validation, and JCS for schema identification. The aim is to establish clear rules for schema semantics, identification, validation, and derivation, ultimately making the Neume network more efficient, scalable, and resilient.

---

Replaces #14.