Skip to content

Commit db03872

Browse files
committed
doc: update for cargo-rdme
1 parent 5604e3d commit db03872

2 files changed

Lines changed: 277 additions & 60 deletions

File tree

README.md

Lines changed: 139 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,99 @@
11
<div align="center"><img src="assets/logo.svg" width="250"/></div>
22

3-
[![Crates.io Version](https://img.shields.io/crates/v/gene?style=for-the-badge)](https://crates.io/crates/gene)
4-
[![GitHub Workflow Status (with event)](https://img.shields.io/github/actions/workflow/status/0xrawsec/gene-rs/ci.yml?style=for-the-badge&logo=github)](https://github.com/kunai-project/gene-rs/actions)
5-
![Crates.io MSRV](https://img.shields.io/crates/msrv/gene?style=for-the-badge)
3+
<!-- cargo-rdme start -->
4+
5+
# Gene - High-Performance Event Scanning and Filtering Engine
66

7+
[![Crates.io Version](https://img.shields.io/crates/v/gene?style=for-the-badge)](https://crates.io/crates/gene)
78
[![Documentation](https://img.shields.io/badge/docs-gene-blue.svg?style=for-the-badge&logo=docsdotrs)](https://docs.rs/gene)
89
[![Documentation](https://img.shields.io/badge/docs-gene_derive-purple.svg?style=for-the-badge&logo=docsdotrs)](https://docs.rs/gene_derive)
10+
![Crates.io MSRV](https://img.shields.io/crates/msrv/gene?style=for-the-badge)
11+
![Crates.io License](https://img.shields.io/crates/l/gene?style=for-the-badge&color=green)
12+
13+
## Project Overview
14+
15+
**Gene** is a Rust implementation of the [original Gene project](https://github.com/0xrawsec/gene) designed
16+
for high-performance event scanning and filtering. Built primarily to power the
17+
[Kunai](https://github.com/kunai-project/kunai) security monitoring system, Gene provides a flexible
18+
and efficient rule-based engine for processing structured log events.
19+
20+
### Purpose
21+
- Embeddable security event scanning engine
22+
- High-throughput log processing and filtering
23+
- Rule-based detection system for security monitoring
24+
25+
### Key Technologies
26+
- **Rule Format**: YAML-based rule definitions for easy authoring
27+
- **Pattern Matching**: Advanced field matching with XPath-like syntax
28+
- **Performance**: Optimized for low-latency, high-volume event processing
29+
30+
### Target Audience
31+
- Security engineers building detection systems
32+
- DevOps teams implementing log monitoring
33+
- Rust developers needing event processing capabilities
34+
35+
## Installation
36+
37+
Add Gene to your project:
938

10-
# Description
39+
```bash
40+
cargo add gene
41+
cargo add gene_derive
42+
```
43+
44+
## Quickstart
45+
46+
```rust
47+
use gene::{Compiler, Engine, Event, FieldGetter, FieldValue};
48+
use gene_derive::{Event, FieldGetter};
49+
50+
// 1. Define your event structure
51+
#[derive(Event, FieldGetter)]
52+
#[event(id = 1, source = "syslog".into())]
53+
struct LogEvent {
54+
message: String,
55+
severity: u8,
56+
}
57+
58+
// 2. Create compiler and load rules
59+
let mut compiler = Compiler::new();
60+
compiler.load_rules_from_str(
61+
r#"
62+
name: high.severity
63+
matches:
64+
$sev: .severity > '5'
65+
condition: $sev"#
66+
).unwrap();
67+
68+
// 3. Build the scanning engine
69+
let mut engine = Engine::try_from(compiler).unwrap();
70+
71+
// 4. Scan events
72+
let event = LogEvent {
73+
message: "Critical error".to_string(),
74+
severity: 8,
75+
};
76+
77+
let scan_result = engine.scan(&event).unwrap();
78+
if scan_result.includes_detection("high.severity") {
79+
println!("High severity event detected!");
80+
}
81+
```
82+
83+
## Core Concepts
84+
85+
| Concept | Description |
86+
|---------|-------------|
87+
| **Events** | Structured data representing log entries or system events |
88+
| **Rules** | Pattern matching and condition evaluation definitions |
89+
| **Matches** | Field extraction and pattern matching expressions |
90+
| **Conditions** | Boolean logic combining match results |
91+
| **Decisions** | Include/exclude logic for scan results |
92+
| **Templates** | Dynamic rule configuration through variable substitution |
1193

12-
This project is a Rust implementation of the [Gene project](https://github.com/0xrawsec/gene) initially
13-
written in Go. The main objective of this project is to embed a security event scanning engine to
14-
[Kunai](https://github.com/kunai-project/kunai). Even though it has been built for a specific use case,
15-
the code in this library is completely re-usable for other log scanning purposes.
94+
## Rule Format
1695

17-
This re-implementation was also the occasion to completely rework the rule format, to
18-
make it simpler, better structured and easier to write. It is now using the [YAML](https://yaml.org/) document
19-
format to encode rule information.
96+
Gene uses YAML for rule definitions, providing a clean and structured format:
2097

2198
```yaml
2299
name: mimic.kthread
@@ -28,49 +105,75 @@ meta:
28105
- tries to catch binaries masquerading kernel threads
29106
match-on:
30107
events:
31-
# we match kunai events execve and execve_script
32-
kunai: [1,2]
108+
kunai: [1,2] # Match specific event types
33109
matches:
34-
# 0x200000 is the flag for KTHREAD
35110
$task_is_kthread: .info.task.flags &= '0x200000'
36-
# common kthread names
37111
$kthread_names: .info.task.name ~= '^(kworker)'
38-
# if task is NOT a KTHREAD but we have a name that looks like one
39112
condition: not $task_is_kthread and $kthread_names
40113
severity: 10
41114
```
42115
43-
# Benchmarks
116+
### Rule Components
44117
45-
Even though the following benchmarks were made with **real** detection rules and **real security events**
46-
performances are indicative. I would say that the throughput is not bad, at least to fulfill the main objective of
47-
this project. The most important aspect being that this library does not become the bottleneck of the
48-
program in which it is embedded.
118+
- **`name`**: Unique rule identifier
119+
- **`meta`**: Metadata including tags, attack IDs, authors
120+
- **`match-on`**: Event type filtering
121+
- **`matches`**: Field extraction and pattern matching
122+
- **`condition`**: Boolean logic for detection
123+
- **`severity`**: Numerical severity level
49124

50-
To determine whether this library might be a bottleneck for your application, try to evaluate the number
51-
of events you want to scan per second and see if it is above the processing throughput.
125+
## Features
52126

53-
## Engine loaded with hundred-ish rules (1 thread)
127+
### High Performance
128+
- Optimized for low-latency event processing
129+
- Efficient pattern matching algorithms
130+
- Minimal memory overhead
54131

55-
```
56-
Number of scanned events: 1001600 -> 1327.72 MB
57-
Number of loaded rules: 127
58-
Scan duration: 1.279534249s -> 1037.66 MB/s -> 782784.83 events/s
59-
Number of detections: 550
60-
```
132+
### Flexible Matching
133+
- XPath-like field access (`.field.subfield`)
134+
- Regular expression support (`~=` operator)
135+
- Bitwise operations (`&=`, `|=`, etc.)
136+
- Comparison operators (`>`, `<`, `==`, etc.)
61137

62-
## Engine loaded with thousand-ish rules (1 thread)
138+
### Advanced Capabilities
139+
- **Rule Dependencies**: Chain rules together for complex detection logic
140+
- **Template System**: Dynamic rule configuration with variable substitution
141+
- **Metadata Support**: Rich metadata including MITRE ATT&CK mappings
142+
- **Decision System**: Fine-grained control over event inclusion/exclusion
63143

64-
```
65-
Number of scanned events: 1001600 -> 1327.72 MB
66-
Number of loaded rules: 1016
67-
Scan duration: 9.535205107s -> 139.24 MB/s -> 105042.31 events/s
68-
Number of detections: 550
144+
## Performance Benchmarks
145+
146+
Benchmarks conducted with real detection rules and security events:
147+
148+
### Hundred-ish Rules (127 rules)
149+
```text
150+
Number of scanned events: 1,001,600 (1,327.72 MB)
151+
Scan duration: 1.28s
152+
Throughput: 1,037.66 MB/s | 782,784.83 events/s
153+
Detections: 550
69154
```
70155

156+
### Thousand-ish Rules (1,016 rules)
157+
```text
158+
Number of scanned events: 1,001,600 (1,327.72 MB)
159+
Scan duration: 9.54s
160+
Throughput: 139.24 MB/s | 105,042.31 events/s
161+
Detections: 550
162+
```
71163

164+
> **Note**: Performance scales with rule complexity. These benchmarks demonstrate
165+
> that Gene remains efficient even with large rule sets, avoiding bottleneck issues
166+
> in embedded applications.
72167

168+
### Contributing
73169

170+
- Report issues on [GitHub](https://github.com/kunai-project/gene-rs/issues)
171+
- Submit pull requests with clear descriptions
172+
- Follow Rust API guidelines and documentation standards
173+
- Maintain `cargo test` and `cargo clippy` cleanliness
74174

175+
## License
75176

177+
Gene is licensed under the **GPL-3.0** - see the `LICENSE` file for details.
76178

179+
<!-- cargo-rdme end -->

0 commit comments

Comments
 (0)