Skip to content

Commit ac06166

Browse files
qjeromeqjerome
authored
Merge pull request #22 from kunai-project/doc-undocumented-apis
improve documentation
2 parents 10cfe4b + db03872 commit ac06166

6 files changed

Lines changed: 832 additions & 62 deletions

File tree

‎README.md‎

Lines changed: 139 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,99 @@
11
<div align="center"><img src="assets/logo.svg" width="250"/></div>
22

3-
[![Crates.io Version](https://img.shields.io/crates/v/gene?style=for-the-badge)](https://crates.io/crates/gene)
4-
[![GitHub Workflow Status (with event)](https://img.shields.io/github/actions/workflow/status/0xrawsec/gene-rs/ci.yml?style=for-the-badge&logo=github)](https://github.com/kunai-project/gene-rs/actions)
5-
![Crates.io MSRV](https://img.shields.io/crates/msrv/gene?style=for-the-badge)
3+
<!-- cargo-rdme start -->
4+
5+
# Gene - High-Performance Event Scanning and Filtering Engine
66

7+
[![Crates.io Version](https://img.shields.io/crates/v/gene?style=for-the-badge)](https://crates.io/crates/gene)
78
[![Documentation](https://img.shields.io/badge/docs-gene-blue.svg?style=for-the-badge&logo=docsdotrs)](https://docs.rs/gene)
89
[![Documentation](https://img.shields.io/badge/docs-gene_derive-purple.svg?style=for-the-badge&logo=docsdotrs)](https://docs.rs/gene_derive)
10+
![Crates.io MSRV](https://img.shields.io/crates/msrv/gene?style=for-the-badge)
11+
![Crates.io License](https://img.shields.io/crates/l/gene?style=for-the-badge&color=green)
12+
13+
## Project Overview
14+
15+
**Gene** is a Rust implementation of the [original Gene project](https://github.com/0xrawsec/gene) designed
16+
for high-performance event scanning and filtering. Built primarily to power the
17+
[Kunai](https://github.com/kunai-project/kunai) security monitoring system, Gene provides a flexible
18+
and efficient rule-based engine for processing structured log events.
19+
20+
### Purpose
21+
- Embeddable security event scanning engine
22+
- High-throughput log processing and filtering
23+
- Rule-based detection system for security monitoring
24+
25+
### Key Technologies
26+
- **Rule Format**: YAML-based rule definitions for easy authoring
27+
- **Pattern Matching**: Advanced field matching with XPath-like syntax
28+
- **Performance**: Optimized for low-latency, high-volume event processing
29+
30+
### Target Audience
31+
- Security engineers building detection systems
32+
- DevOps teams implementing log monitoring
33+
- Rust developers needing event processing capabilities
34+
35+
## Installation
36+
37+
Add Gene to your project:
938

10-
# Description
39+
```bash
40+
cargo add gene
41+
cargo add gene_derive
42+
```
43+
44+
## Quickstart
45+
46+
```rust
47+
use gene::{Compiler, Engine, Event, FieldGetter, FieldValue};
48+
use gene_derive::{Event, FieldGetter};
49+
50+
// 1. Define your event structure
51+
#[derive(Event, FieldGetter)]
52+
#[event(id = 1, source = "syslog".into())]
53+
struct LogEvent {
54+
message: String,
55+
severity: u8,
56+
}
57+
58+
// 2. Create compiler and load rules
59+
let mut compiler = Compiler::new();
60+
compiler.load_rules_from_str(
61+
r#"
62+
name: high.severity
63+
matches:
64+
$sev: .severity > '5'
65+
condition: $sev"#
66+
).unwrap();
67+
68+
// 3. Build the scanning engine
69+
let mut engine = Engine::try_from(compiler).unwrap();
70+
71+
// 4. Scan events
72+
let event = LogEvent {
73+
message: "Critical error".to_string(),
74+
severity: 8,
75+
};
76+
77+
let scan_result = engine.scan(&event).unwrap();
78+
if scan_result.includes_detection("high.severity") {
79+
println!("High severity event detected!");
80+
}
81+
```
82+
83+
## Core Concepts
84+
85+
| Concept | Description |
86+
|---------|-------------|
87+
| **Events** | Structured data representing log entries or system events |
88+
| **Rules** | Pattern matching and condition evaluation definitions |
89+
| **Matches** | Field extraction and pattern matching expressions |
90+
| **Conditions** | Boolean logic combining match results |
91+
| **Decisions** | Include/exclude logic for scan results |
92+
| **Templates** | Dynamic rule configuration through variable substitution |
1193

12-
This project is a Rust implementation of the [Gene project](https://github.com/0xrawsec/gene) initially
13-
written in Go. The main objective of this project is to embed a security event scanning engine to
14-
[Kunai](https://github.com/kunai-project/kunai). Even though it has been built for a specific use case,
15-
the code in this library is completely re-usable for other log scanning purposes.
94+
## Rule Format
1695

17-
This re-implementation was also the occasion to completely rework the rule format, to
18-
make it simpler, better structured and easier to write. It is now using the [YAML](https://yaml.org/) document
19-
format to encode rule information.
96+
Gene uses YAML for rule definitions, providing a clean and structured format:
2097

2198
```yaml
2299
name: mimic.kthread
@@ -28,49 +105,75 @@ meta:
28105
- tries to catch binaries masquerading kernel threads
29106
match-on:
30107
events:
31-
# we match kunai events execve and execve_script
32-
kunai: [1,2]
108+
kunai: [1,2] # Match specific event types
33109
matches:
34-
# 0x200000 is the flag for KTHREAD
35110
$task_is_kthread: .info.task.flags &= '0x200000'
36-
# common kthread names
37111
$kthread_names: .info.task.name ~= '^(kworker)'
38-
# if task is NOT a KTHREAD but we have a name that looks like one
39112
condition: not $task_is_kthread and $kthread_names
40113
severity: 10
41114
```
42115
43-
# Benchmarks
116+
### Rule Components
44117
45-
Even though the following benchmarks were made with **real** detection rules and **real security events**
46-
performances are indicative. I would say that the throughput is not bad, at least to fulfill the main objective of
47-
this project. The most important aspect being that this library does not become the bottleneck of the
48-
program in which it is embedded.
118+
- **`name`**: Unique rule identifier
119+
- **`meta`**: Metadata including tags, attack IDs, authors
120+
- **`match-on`**: Event type filtering
121+
- **`matches`**: Field extraction and pattern matching
122+
- **`condition`**: Boolean logic for detection
123+
- **`severity`**: Numerical severity level
49124

50-
To determine whether this library might be a bottleneck for your application, try to evaluate the number
51-
of events you want to scan per second and see if it is above the processing throughput.
125+
## Features
52126

53-
## Engine loaded with hundred-ish rules (1 thread)
127+
### High Performance
128+
- Optimized for low-latency event processing
129+
- Efficient pattern matching algorithms
130+
- Minimal memory overhead
54131

55-
```
56-
Number of scanned events: 1001600 -> 1327.72 MB
57-
Number of loaded rules: 127
58-
Scan duration: 1.279534249s -> 1037.66 MB/s -> 782784.83 events/s
59-
Number of detections: 550
60-
```
132+
### Flexible Matching
133+
- XPath-like field access (`.field.subfield`)
134+
- Regular expression support (`~=` operator)
135+
- Bitwise operations (`&=`, `|=`, etc.)
136+
- Comparison operators (`>`, `<`, `==`, etc.)
61137

62-
## Engine loaded with thousand-ish rules (1 thread)
138+
### Advanced Capabilities
139+
- **Rule Dependencies**: Chain rules together for complex detection logic
140+
- **Template System**: Dynamic rule configuration with variable substitution
141+
- **Metadata Support**: Rich metadata including MITRE ATT&CK mappings
142+
- **Decision System**: Fine-grained control over event inclusion/exclusion
63143

64-
```
65-
Number of scanned events: 1001600 -> 1327.72 MB
66-
Number of loaded rules: 1016
67-
Scan duration: 9.535205107s -> 139.24 MB/s -> 105042.31 events/s
68-
Number of detections: 550
144+
## Performance Benchmarks
145+
146+
Benchmarks conducted with real detection rules and security events:
147+
148+
### Hundred-ish Rules (127 rules)
149+
```text
150+
Number of scanned events: 1,001,600 (1,327.72 MB)
151+
Scan duration: 1.28s
152+
Throughput: 1,037.66 MB/s | 782,784.83 events/s
153+
Detections: 550
69154
```
70155

156+
### Thousand-ish Rules (1,016 rules)
157+
```text
158+
Number of scanned events: 1,001,600 (1,327.72 MB)
159+
Scan duration: 9.54s
160+
Throughput: 139.24 MB/s | 105,042.31 events/s
161+
Detections: 550
162+
```
71163

164+
> **Note**: Performance scales with rule complexity. These benchmarks demonstrate
165+
> that Gene remains efficient even with large rule sets, avoiding bottleneck issues
166+
> in embedded applications.
72167

168+
### Contributing
73169

170+
- Report issues on [GitHub](https://github.com/kunai-project/gene-rs/issues)
171+
- Submit pull requests with clear descriptions
172+
- Follow Rust API guidelines and documentation standards
173+
- Maintain `cargo test` and `cargo clippy` cleanliness
74174

175+
## License
75176

177+
Gene is licensed under the **GPL-3.0** - see the `LICENSE` file for details.
76178

179+
<!-- cargo-rdme end -->

‎gene/src/event.rs‎

Lines changed: 108 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,20 +7,125 @@ use std::{
77

88
use crate::{FieldValue, XPath};
99

10-
/// Trait representing a log event
10+
/// Trait representing a log event that can be scanned by the engine.
11+
///
12+
/// Events provide access to their unique identifier, source, and field values
13+
/// through the [`FieldGetter`] trait. This trait is typically derived using
14+
/// the [`Event`] derive macro from `gene_derive`.
15+
///
16+
/// # Examples
17+
///
18+
/// ```
19+
/// use gene::{FieldValue, Event, FieldGetter};
20+
/// use gene_derive::{Event, FieldGetter};
21+
/// use std::borrow::Cow;
22+
///
23+
/// // Simple event with derive macro
24+
/// #[derive(Event, FieldGetter)]
25+
/// #[event(id = 42, source = "syslog".into())]
26+
/// struct SyslogEvent {
27+
/// message: String,
28+
/// severity: u8,
29+
/// }
30+
/// ```
1131
pub trait Event<'event>: FieldGetter<'event> {
32+
/// Returns the unique identifier for this event.
33+
///
34+
/// The ID is used by rules to determine which events they apply to
35+
/// through the `match-on` directive.
1236
fn id(&self) -> i64;
37+
38+
/// Returns the source of this event.
39+
///
40+
/// The source is used by rules to determine which events they apply to
41+
/// through the `match-on` directive.
1342
fn source(&self) -> Cow<'_, str>;
1443
}
1544

16-
/// Trait representing a structure we can fetch field values
17-
/// from a [`XPath`]
45+
/// Trait for fetching field values from structured data using XPath-like paths.
46+
///
47+
/// Implementors of this trait provide access to their fields through two methods:
48+
/// - `get_from_path`: The primary interface using [`XPath`] expressions
49+
/// - `get_from_iter`: A lower-level interface using path segment iterators
50+
///
51+
/// This trait is typically implemented automatically via the [`FieldGetter`] derive macro,
52+
/// but can be implemented manually for custom data structures.
53+
///
54+
/// # Examples
55+
///
56+
/// ```
57+
/// use gene::{FieldGetter, FieldValue};
58+
/// use std::net::IpAddr;
59+
///
60+
/// struct NetworkEvent {
61+
/// source_ip: IpAddr,
62+
/// destination_ip: IpAddr,
63+
/// port: u16,
64+
/// }
65+
///
66+
/// impl<'f> FieldGetter<'f> for NetworkEvent {
67+
/// fn get_from_iter(
68+
/// &'f self,
69+
/// mut i: core::slice::Iter<'_, std::string::String>,
70+
/// ) -> Option<FieldValue<'f>> {
71+
/// match i.next().map(|s| s.as_str()) {
72+
/// Some("source_ip") => Some(self.source_ip.to_string().into()),
73+
/// Some("destination_ip") => Some(self.destination_ip.to_string().into()),
74+
/// Some("port") => Some(self.port.into()),
75+
/// _ => None,
76+
/// }
77+
/// }
78+
/// }
79+
/// ```
80+
///
81+
/// # Field Value Types
82+
///
83+
/// The trait returns [`FieldValue`] enum which can represent:
84+
/// - Primitive types (numbers, booleans, strings)
85+
/// - Complex types (IP addresses, paths, etc.)
86+
/// - Collections (vectors, hash maps)
87+
/// - Optional/nested values
88+
///
89+
/// Rules use these field values for pattern matching and condition evaluation.
1890
pub trait FieldGetter<'field> {
91+
/// Gets a field value using an [`XPath`] expression.
92+
///
93+
/// This is the primary method for field access and is called by the engine
94+
/// when evaluating rule conditions. The default implementation delegates
95+
/// to `get_from_iter` for convenience.
96+
///
97+
/// # Arguments
98+
///
99+
/// * `path` - The XPath expression identifying the field to retrieve
100+
///
101+
/// # Returns
102+
///
103+
/// * `Some(FieldValue)` if the field exists and can be accessed
104+
/// * `None` if the field does not exist or cannot be accessed
105+
///
106+
/// # Notes
107+
///
108+
/// Most implementations should rely on the default implementation and
109+
/// override [`Self::get_from_iter`] instead of overriding this method.
19110
#[inline]
20111
fn get_from_path(&'field self, path: &XPath) -> Option<FieldValue<'field>> {
21112
self.get_from_iter(path.iter_segments())
22113
}
23114

115+
/// Gets a field value using an iterator of path segments.
116+
///
117+
/// This lower-level method provides direct access to path segments for
118+
/// implementors who need more control over path processing. It's called
119+
/// by the default implementation of `get_from_path`.
120+
///
121+
/// # Arguments
122+
///
123+
/// * `i` - An iterator over the path segments
124+
///
125+
/// # Returns
126+
///
127+
/// * `Some(FieldValue)` if the field exists and can be accessed
128+
/// * `None` if the field does not exist or cannot be accessed
24129
fn get_from_iter(
25130
&'field self,
26131
i: core::slice::Iter<'_, std::string::String>,

0 commit comments

Comments
 (0)