Skip to content

Commit fdf172e

Browse files
committed
add graylog reporting infos
1 parent 0d9b15b commit fdf172e

7 files changed

+177
-4
lines changed

README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -159,7 +159,7 @@ If you want to get a (free) token for your systems - feel free to contact us at:
159159

160160
A simple script that follows the content of a specific log-file and parses abuser information from it.
161161

162-
See: [Report Script](https://github.com/O-X-L/risk-db/blob/latest/report_script/README.md)
162+
See: [Report Script](https://github.com/O-X-L/risk-db/blob/latest/reporting/README.md)
163163

164164

165165
#### Fail2Ban

reporting/Graylog.md

+173
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# Reporting via Graylog Alerts
2+
3+
We can create a Graylog Alert Notification to Report Abusers to this Risk-Database.
4+
5+
You can find an example on how to split HAProxy logs into different fields here: [gist.github.com](https://gist.github.com/superstes/a2f6c5d855857e1f10dcb51255fe08c6#haproxy-split) (*via Pipeline Rules*)
6+
7+
## API Service
8+
9+
As Graylog has no option to add advanced filters for the data sent by the notifications, we will have to add a minimal service to do so.
10+
11+
1. Add the API Service Script: (File: `/usr/local/bin/notification-api.py`)
12+
13+
```python3
14+
#!/usr/bin/env python3
15+
16+
# Abuse Reporting System: https://github.com/O-X-L/risk-db
17+
18+
from time import sleep
19+
20+
from waitress import serve
21+
from flask import Flask, request, jsonify, make_response
22+
from requests import post as http_post
23+
from requests.exceptions import HTTPError, SSLError
24+
25+
DEBUG = False
26+
PORT = 8000
27+
RISK_DB_TOKEN = 'XXX'
28+
29+
# for internal proxy servers
30+
# proxy_server = {
31+
# 'http': 'http://{{ proxy_server }}:{{ proxy_server_port }}',
32+
# 'https': 'http://{{ proxy_server }}:{{ proxy_server_port }}',
33+
# }
34+
proxy_server = None
35+
36+
app = Flask(__name__)
37+
38+
39+
@app.route('/report-abuse/haproxy', methods=['POST'])
40+
def report_abuse_haproxy():
41+
unique_list = []
42+
43+
for log in request.json['backlog']:
44+
try:
45+
# todo: update the field names to the ones you use
46+
log = log['fields']
47+
48+
# todo: add additional filters here and 'continue' if a log should be skipped
49+
50+
ip = log['haproxy_client']
51+
52+
# making sure we do not report a client twice (every minute)
53+
if ip in unique_list:
54+
continue
55+
56+
unique_list.append(ip)
57+
58+
cause = 'probe'
59+
# todo: add conditions to decide on what category to report on
60+
if str(log['xyz']) != '1':
61+
cause = 'bot'
62+
63+
elif str(log['haproxy_status']) == '429':
64+
cause = 'rate'
65+
66+
ua = log['haproxy_user_agent'][:50]
67+
68+
print(f'REPORTING: {ip} because of {cause} ("{ua}")')
69+
http_post(
70+
url='https://risk.oxl.app/api/report',
71+
proxies=proxy_server,
72+
headers={'Token': RISK_DB_TOKEN},
73+
json={
74+
'ip': ip,
75+
'cat': cause,
76+
'cmt': ua,
77+
},
78+
)
79+
80+
if response.status_code != 200 and DEBUG:
81+
print('Abuse-Report failed', response.status_code)
82+
83+
sleep(1)
84+
85+
except (KeyError, HTTPError, SSLError):
86+
continue
87+
88+
return make_response(jsonify({}), 200)
89+
90+
91+
if __name__ == '__main__':
92+
serve(app, host='127.0.0.1', port=PORT)
93+
```
94+
95+
2. Add a python3 virtual environment:
96+
97+
```bash
98+
apt install python3-virtualenv
99+
python3 -m virtualenv /var/local/graylog-notification-api/venv
100+
source /var/local/graylog-notification-api/venv/bin/activate
101+
pip install requests flask waitress
102+
```
103+
104+
3. Add a systemd service to run the api script: (File: `/etc/systemd/system/graylog-notification-api.service`)
105+
106+
```
107+
[Unit]
108+
Description=Graylog Notification API Service
109+
110+
[Service]
111+
Type=simple
112+
User=graylog
113+
Group=graylog
114+
Environment=PYTHONUNBUFFERED=1
115+
ExecStart=/bin/bash -c "source /var/local/graylog-notification-api/venv/bin/activate && python3 /usr/local/bin/notification-api.py"
116+
117+
StandardOutput=journal
118+
StandardError=journal
119+
SyslogIdentifier=notification-api
120+
Restart=on-failure
121+
RestartSec=10s
122+
TimeoutStopSec=30s
123+
124+
[Install]
125+
WantedBy=multi-user.target
126+
```
127+
128+
4. Enable and start the service:
129+
130+
```bash
131+
systemctl daemon-reload
132+
systemctl start graylog-notification-api.service
133+
systemctl enable graylog-notification-api.service
134+
```
135+
136+
----
137+
138+
## Graylog Alerts
139+
140+
### Create an Alert-Notification
141+
142+
`https://<SERVER>/alerts/notifications`
143+
144+
* **Title**: `Report Abuse - HAProxy`
145+
* **Notification Type**: `HTTP Notification`
146+
* **URL**: `http://127.0.0.1:8000/report-abuse/haproxy`
147+
148+
149+
### Create an Alert-Event
150+
151+
`https://<SERVER>/alerts/definitions`
152+
153+
**Event Details**:
154+
155+
* **Title**: `HAProxy Abuse`
156+
* **Priority**: `Low`
157+
158+
**Condition**:
159+
160+
* **Condition Type**: `Filter & Aggregation`
161+
* **Streams**: Select your HAProxy Access-Log stream
162+
* **Search Query**: Filter Logs to only include blocks of your security filters. Also exclude your `safe-ips` and so on
163+
* **Search within the last**: 1 minute
164+
* **Execute search every**: 1 minute
165+
* **Create Events for Definition if...**: `Aggregation of results reaches a threshold`
166+
* **Execute search every**: 1 minute
167+
* **If**: `count()` **is** `>` **Threshold** `0`
168+
169+
**Notifications**:
170+
171+
* **Choose Notification**: `Report Abuse - HAProxy`
172+
* **Grace Period**: Disable
173+
* **Message Backlog**: 500 (duplicates will be filtered by the API-service)

report_script/README.md reporting/README.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Note: Bash regex does not support PCRE like `\d`, `\s` nor non-greedy `*?`
66

77
You can run this script as systemd service:
88

9-
* Example: [report_log_watcher.service](https://github.com/O-X-L/risk-db/blob/latest/report_script/log_watcher_APP.service)
9+
* Example: [report_log_watcher.service](https://github.com/O-X-L/risk-db/blob/latest/reporting/log_watcher_APP.service)
1010
* Copy the scripts to `/usr/local/bin/` (*or wherever you want them to be*)
1111
* Enable/Start:
1212

@@ -45,7 +45,7 @@ See the scripts for examples.
4545

4646
### NFTables Firewall
4747

48-
See: [Report Script - NFTables](https://github.com/O-X-L/risk-db/blob/latest/report_script/log_watcher_nftables.sh)
48+
See: [Report Script - NFTables](https://github.com/O-X-L/risk-db/blob/latest/reporting/log_watcher_nftables.sh)
4949

5050
#### Logs
5151

@@ -94,7 +94,7 @@ Content:
9494

9595
### HAProxy
9696

97-
See: [Report Script - HAProxy](https://github.com/O-X-L/risk-db/blob/latest/report_script/log_watcher_haproxy.sh)
97+
See: [Report Script - HAProxy](https://github.com/O-X-L/risk-db/blob/latest/reporting/log_watcher_haproxy.sh)
9898

9999
#### Logs
100100

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.

0 commit comments

Comments
 (0)