Merge pull request #62 from timescale/feat/lynxdb-backend

feat(convert): add LynxDB backend

Author: mostafa

README.md

@@ -173,6 +173,15 @@ rsigma convert rules/ -t postgres -O table=okta_events -O json_field=data -O tim
 # Sliding window correlation format (per-row detection using window functions)
 rsigma convert rules/ -t postgres -f sliding_window
 
+# Convert to LynxDB search queries
+rsigma convert rules/ -t lynxdb
+
+# Convert to LynxDB with a pipeline (custom index)
+rsigma convert rules/ -t lynxdb -p pipeline.yml
+
+# LynxDB minimal format (search expression only, for the API q parameter)
+rsigma convert rules/ -t lynxdb -f minimal
+
 # List available conversion backends
 rsigma list-targets
 
@@ -260,7 +269,7 @@ From there, the AST can go in three directions depending on what you need:
 
 - **Evaluation:** `rsigma-eval` compiles rules into optimized matchers (`compiler.rs`), runs stateless detection through `Engine`, and tracks stateful correlation (`correlation.rs`: sliding windows, group-by, chaining, suppression) across events. Processing pipelines handle field mapping, transformations, conditions, and finalizers before compilation. Events are accessed through a trait with implementations for JSON, key-value, and plain text.
 
-- **Conversion:** `rsigma-convert` transforms rules into backend-native query strings through a pluggable `Backend` trait. A condition walker traverses the AST and delegates to the backend for each node. `TextQueryConfig` exposes ~90 configuration fields for text-based backends. The PostgreSQL/TimescaleDB backend is the primary concrete implementation, generating SQL for historical threat hunting.
+- **Conversion:** `rsigma-convert` transforms rules into backend-native query strings through a pluggable `Backend` trait. A condition walker traverses the AST and delegates to the backend for each node. `TextQueryConfig` exposes ~90 configuration fields for text-based backends. Concrete implementations include PostgreSQL/TimescaleDB (SQL for historical threat hunting) and LynxDB (SPL2-compatible search queries for log analytics).
 
 - **Editor support:** `rsigma-lsp` provides an LSP server over stdio (via `tower-lsp`) with real-time diagnostics (lint + parse + compile errors), completions, hover documentation, document symbols, and code actions. Works with VSCode, Neovim, Helix, Zed, and any LSP-capable editor.
 
@@ -320,8 +329,9 @@ Feature-gated items are marked with \* in the diagram.
     │                         │   │  backends/ ──>      │   │  Helix, Zed, ...   │
     │  correlation.rs ──>     │   │    TextQueryTest,   │   └────────────────────┘
     │    sliding windows,     │   │    PostgreSQL/      │
-    │    group-by, chaining,  │   │    TimescaleDB      │
-    │    suppression, events  │   └─────────────────────┘
+    │    group-by, chaining,  │   │    TimescaleDB,     │
+    │    suppression, events  │   │    LynxDB           │
+    │                         │   └─────────────────────┘
     │                         │
     │  rsigma.* custom        │
     │    attributes           │

crates/rsigma-cli/src/commands/convert.rs

@@ -11,13 +11,14 @@ fn get_backend(
         "postgres" | "postgresql" | "pg" => {
             Box::new(rsigma_convert::backends::postgres::PostgresBackend::from_options(options))
         }
+        "lynxdb" => Box::new(rsigma_convert::backends::lynxdb::LynxDbBackend::new()),
         "test" => Box::new(rsigma_convert::backends::test::TextQueryTestBackend::new()),
         "test_mandatory_pipeline" => {
             Box::new(rsigma_convert::backends::test::MandatoryPipelineTestBackend::new())
         }
         _ => {
             eprintln!("Unknown target: {target}");
-            eprintln!("Available targets: postgres, test");
+            eprintln!("Available targets: postgres, lynxdb, test");
             process::exit(1);
         }
     }
@@ -100,6 +101,7 @@ pub(crate) fn cmd_convert(
 pub(crate) fn cmd_list_targets() {
     println!("Available conversion targets:");
     println!("  postgres  - PostgreSQL/TimescaleDB (aliases: postgresql, pg)");
+    println!("  lynxdb    - LynxDB log analytics engine");
     println!("  test      - Backend-neutral test backend");
 }
 

crates/rsigma-cli/tests/cli_convert.rs

@@ -217,7 +217,7 @@ fn convert_invalid_target() {
     assert!(!output.status.success());
     assert_snapshot!(String::from_utf8_lossy(&output.stderr), @"
     Unknown target: nonexistent_backend
-    Available targets: postgres, test
+    Available targets: postgres, lynxdb, test
     ");
 }
 
@@ -294,6 +294,7 @@ fn list_targets() {
     assert_snapshot!(String::from_utf8_lossy(&output.stdout), @"
     Available conversion targets:
       postgres  - PostgreSQL/TimescaleDB (aliases: postgresql, pg)
+      lynxdb    - LynxDB log analytics engine
       test      - Backend-neutral test backend
     ");
 }
@@ -324,6 +325,6 @@ fn list_formats_invalid_target() {
     assert!(!output.status.success());
     assert_snapshot!(String::from_utf8_lossy(&output.stderr), @"
     Unknown target: nonexistent
-    Available targets: postgres, test
+    Available targets: postgres, lynxdb, test
     ");
 }

crates/rsigma-convert/README.md

@@ -17,13 +17,15 @@ The crate provides a generic conversion framework that any backend can plug into
 - **Deferred expressions** through the `DeferredExpression` trait and `DeferredTextExpression` for backends that need post-query appendages (e.g. Splunk `| regex`, `| where`).
 - **Test backend** with `TextQueryTestBackend` and `MandatoryPipelineTestBackend` for backend-neutral foundation testing.
 - **PostgreSQL/TimescaleDB backend** with native `ILIKE`, regex (`~*`), CIDR (`inet`/`cidr`), full-text search (`tsvector`/`tsquery`), JSONB field access, correlation via CTEs and window functions, and TimescaleDB-specific output formats (continuous aggregates, `time_bucket` queries, view generation).
+- **LynxDB backend** generating SPL2-compatible `FROM <index> | search ...` queries with glob wildcards, deferred `| where` clauses for regex and CIDR, `CASE()` case-sensitive matching, and correct parenthesization for LynxDB's non-standard boolean precedence (`NOT > OR > AND`).
 
 ## Backends
 
 | Backend | Target names | Description |
 |---------|-------------|-------------|
 | Test | `test` | Backend-neutral text queries for foundation testing |
 | PostgreSQL | `postgres`, `postgresql`, `pg` | Native PostgreSQL SQL with TimescaleDB support |
+| LynxDB | `lynxdb` | SPL2-compatible search queries for LynxDB log analytics engine |
 
 ## Usage
 
@@ -89,6 +91,61 @@ for result in &output.queries {
 }
 ```
 
+### LynxDB backend
+
+```rust
+use rsigma_parser::parse_sigma_yaml;
+use rsigma_convert::{convert_collection, Backend};
+use rsigma_convert::backends::lynxdb::LynxDbBackend;
+
+let yaml = r#"
+title: Detect Whoami
+logsource:
+    category: process_creation
+    product: windows
+detection:
+    selection:
+        CommandLine|contains: 'whoami'
+    condition: selection
+level: medium
+"#;
+
+let collection = parse_sigma_yaml(yaml).unwrap();
+let backend = LynxDbBackend::new();
+
+let output = convert_collection(&backend, &collection, &[], "default").unwrap();
+for result in &output.queries {
+    for query in &result.queries {
+        println!("{query}");
+        // Output: FROM main | search CommandLine=*"whoami"*
+    }
+}
+```
+
+### LynxDB output formats
+
+| Format | Description |
+|--------|-------------|
+| `default` | Full query with index prefix: `FROM main \| search ...` |
+| `minimal` | Search expression only (no `FROM` prefix), useful for the LynxDB API `q` parameter |
+
+### LynxDB index selection
+
+The target index defaults to `main`. Set it via pipeline state:
+
+```yaml
+# In a pipeline YAML
+transformations:
+  - type: set_state
+    key: index
+    value: security_logs
+```
+
+```bash
+rsigma convert -r rules/ -t lynxdb -p pipeline.yml
+# Output: FROM security_logs | search ...
+```
+
 ### PostgreSQL output formats
 
 | Format | Description |
@@ -231,7 +288,7 @@ For text-based query backends (the vast majority), create a `TextQueryConfig` wi
 3. Override specific methods for backend-specific behavior (e.g. deferred regex for Splunk, SQL-specific CIDR handling for PostgreSQL).
 4. Register your backend in the CLI's `get_backend()` registry.
 
-See `backends/test.rs` for a complete reference implementation and `backends/postgres.rs` for a production backend with SQL-specific overrides.
+See `backends/test.rs` for a complete reference implementation, `backends/postgres.rs` for a production backend with SQL-specific overrides, and `backends/lynxdb/` for a `TextQueryConfig`-based backend with deferred expressions and custom precedence handling.
 
 ## PostgreSQL Backend Details
 
@@ -308,6 +365,47 @@ This matches the nested traversal behavior of the evaluation engine (`rsigma-eva
 rsigma convert -r rules/ -t postgres -O table=okta_events -O json_field=data -O timestamp_field=time
 ```
 
+## LynxDB Backend Details
+
+The LynxDB backend (`LynxDbBackend`) generates SPL2/Lynx Flow queries for the [LynxDB](https://github.com/proximax-storage/lynxdb) log analytics engine. It produces `FROM <index> | search <predicates>` queries with deferred `| where` clauses for operations that LynxDB's search syntax does not natively support.
+
+| Sigma Modifier | LynxDB Query |
+|----------------|-------------|
+| `contains` | `field=*"value"*` |
+| `startswith` | `field="value"*` |
+| `endswith` | `field=*"value"` |
+| `re` | `\| where field=~"pattern"` (deferred) |
+| `cidr` | `\| where cidrmatch("cidr", field)` (deferred) |
+| `cased` (exact) | `field=CASE("value")` |
+| wildcards (`*`) | `field="va*lue"` (glob) |
+| wildcards (`?`) | `\| where field=~"va.lue"` (deferred, converted to regex) |
+| `exists` | `field=*` |
+| `null` | `NOT field=*` |
+| keywords | `"value"` (unbound search) |
+
+### Boolean precedence
+
+LynxDB's parser uses non-standard boolean operator precedence: `NOT > OR > AND`. This differs from most query languages where AND binds tighter than OR. The backend explicitly parenthesizes AND groups to preserve Sigma's intended logic:
+
+```
+Sigma: A AND B OR C    (intended: (A AND B) OR C)
+Query: (A AND B) OR C  (explicit parens prevent misparse as A AND (B OR C))
+```
+
+### Deferred expressions
+
+Regex patterns, CIDR matches, and single-character wildcard (`?`) patterns cannot be expressed in LynxDB's `search` syntax and are instead emitted as `| where` pipeline stages appended after the search clause:
+
+```
+FROM main | search status=500 | where Path=~"/api/.*"
+```
+
+When a detection contains only deferred expressions, the search clause uses `*` (match all) followed by the deferred stages:
+
+```
+FROM main | search * | where SourceIP=~"^10\.0\." | where cidrmatch("192.168.1.0/24", DestIP)
+```
+
 ## License
 
 MIT License.