|
| 1 | +# Reporting via Graylog Alerts |
| 2 | + |
| 3 | +We can create a Graylog Alert Notification to Report Abusers to this Risk-Database. |
| 4 | + |
| 5 | +You can find an example on how to split HAProxy logs into different fields here: [gist.github.com](https://gist.github.com/superstes/a2f6c5d855857e1f10dcb51255fe08c6#haproxy-split) (*via Pipeline Rules*) |
| 6 | + |
| 7 | +## API Service |
| 8 | + |
| 9 | +As Graylog has no option to add advanced filters for the data sent by the notifications, we will have to add a minimal service to do so. |
| 10 | + |
| 11 | +1. Add the API Service Script: (File: `/usr/local/bin/notification-api.py`) |
| 12 | + |
| 13 | + ```python3 |
| 14 | + #!/usr/bin/env python3 |
| 15 | + |
| 16 | + # Abuse Reporting System: https://github.com/O-X-L/risk-db |
| 17 | + |
| 18 | + from time import sleep |
| 19 | + |
| 20 | + from waitress import serve |
| 21 | + from flask import Flask, request, jsonify, make_response |
| 22 | + from requests import post as http_post |
| 23 | + from requests.exceptions import HTTPError, SSLError |
| 24 | + |
| 25 | + DEBUG = False |
| 26 | + PORT = 8000 |
| 27 | + RISK_DB_TOKEN = 'XXX' |
| 28 | + |
| 29 | + # for internal proxy servers |
| 30 | + # proxy_server = { |
| 31 | + # 'http': 'http://{{ proxy_server }}:{{ proxy_server_port }}', |
| 32 | + # 'https': 'http://{{ proxy_server }}:{{ proxy_server_port }}', |
| 33 | + # } |
| 34 | + proxy_server = None |
| 35 | + |
| 36 | + app = Flask(__name__) |
| 37 | + |
| 38 | + |
| 39 | + @app.route('/report-abuse/haproxy', methods=['POST']) |
| 40 | + def report_abuse_haproxy(): |
| 41 | + unique_list = [] |
| 42 | + |
| 43 | + for log in request.json['backlog']: |
| 44 | + try: |
| 45 | + # todo: update the field names to the ones you use |
| 46 | + log = log['fields'] |
| 47 | + |
| 48 | + # todo: add additional filters here and 'continue' if a log should be skipped |
| 49 | + |
| 50 | + ip = log['haproxy_client'] |
| 51 | + |
| 52 | + # making sure we do not report a client twice (every minute) |
| 53 | + if ip in unique_list: |
| 54 | + continue |
| 55 | + |
| 56 | + unique_list.append(ip) |
| 57 | + |
| 58 | + cause = 'probe' |
| 59 | + # todo: add conditions to decide on what category to report on |
| 60 | + if str(log['xyz']) != '1': |
| 61 | + cause = 'bot' |
| 62 | + |
| 63 | + elif str(log['haproxy_status']) == '429': |
| 64 | + cause = 'rate' |
| 65 | + |
| 66 | + ua = log['haproxy_user_agent'][:50] |
| 67 | + |
| 68 | + print(f'REPORTING: {ip} because of {cause} ("{ua}")') |
| 69 | + http_post( |
| 70 | + url='https://risk.oxl.app/api/report', |
| 71 | + proxies=proxy_server, |
| 72 | + headers={'Token': RISK_DB_TOKEN}, |
| 73 | + json={ |
| 74 | + 'ip': ip, |
| 75 | + 'cat': cause, |
| 76 | + 'cmt': ua, |
| 77 | + }, |
| 78 | + ) |
| 79 | + |
| 80 | + if response.status_code != 200 and DEBUG: |
| 81 | + print('Abuse-Report failed', response.status_code) |
| 82 | + |
| 83 | + sleep(1) |
| 84 | + |
| 85 | + except (KeyError, HTTPError, SSLError): |
| 86 | + continue |
| 87 | + |
| 88 | + return make_response(jsonify({}), 200) |
| 89 | + |
| 90 | + |
| 91 | + if __name__ == '__main__': |
| 92 | + serve(app, host='127.0.0.1', port=PORT) |
| 93 | + ``` |
| 94 | + |
| 95 | +2. Add a python3 virtual environment: |
| 96 | + |
| 97 | + ```bash |
| 98 | + apt install python3-virtualenv |
| 99 | + python3 -m virtualenv /var/local/graylog-notification-api/venv |
| 100 | + source /var/local/graylog-notification-api/venv/bin/activate |
| 101 | + pip install requests flask waitress |
| 102 | + ``` |
| 103 | + |
| 104 | +3. Add a systemd service to run the api script: (File: `/etc/systemd/system/graylog-notification-api.service`) |
| 105 | + |
| 106 | + ``` |
| 107 | + [Unit] |
| 108 | + Description=Graylog Notification API Service |
| 109 | + |
| 110 | + [Service] |
| 111 | + Type=simple |
| 112 | + User=graylog |
| 113 | + Group=graylog |
| 114 | + Environment=PYTHONUNBUFFERED=1 |
| 115 | + ExecStart=/bin/bash -c "source /var/local/graylog-notification-api/venv/bin/activate && python3 /usr/local/bin/notification-api.py" |
| 116 | + |
| 117 | + StandardOutput=journal |
| 118 | + StandardError=journal |
| 119 | + SyslogIdentifier=notification-api |
| 120 | + Restart=on-failure |
| 121 | + RestartSec=10s |
| 122 | + TimeoutStopSec=30s |
| 123 | + |
| 124 | + [Install] |
| 125 | + WantedBy=multi-user.target |
| 126 | + ``` |
| 127 | + |
| 128 | +4. Enable and start the service: |
| 129 | + |
| 130 | + ```bash |
| 131 | + systemctl daemon-reload |
| 132 | + systemctl start graylog-notification-api.service |
| 133 | + systemctl enable graylog-notification-api.service |
| 134 | + ``` |
| 135 | + |
| 136 | +---- |
| 137 | + |
| 138 | +## Graylog Alerts |
| 139 | + |
| 140 | +### Create an Alert-Notification |
| 141 | + |
| 142 | +`https://<SERVER>/alerts/notifications` |
| 143 | + |
| 144 | +* **Title**: `Report Abuse - HAProxy` |
| 145 | +* **Notification Type**: `HTTP Notification` |
| 146 | +* **URL**: `http://127.0.0.1:8000/report-abuse/haproxy` |
| 147 | + |
| 148 | + |
| 149 | +### Create an Alert-Event |
| 150 | + |
| 151 | +`https://<SERVER>/alerts/definitions` |
| 152 | + |
| 153 | +**Event Details**: |
| 154 | + |
| 155 | + * **Title**: `HAProxy Abuse` |
| 156 | + * **Priority**: `Low` |
| 157 | + |
| 158 | +**Condition**: |
| 159 | + |
| 160 | + * **Condition Type**: `Filter & Aggregation` |
| 161 | + * **Streams**: Select your HAProxy Access-Log stream |
| 162 | + * **Search Query**: Filter Logs to only include blocks of your security filters. Also exclude your `safe-ips` and so on |
| 163 | + * **Search within the last**: 1 minute |
| 164 | + * **Execute search every**: 1 minute |
| 165 | + * **Create Events for Definition if...**: `Aggregation of results reaches a threshold` |
| 166 | + * **Execute search every**: 1 minute |
| 167 | + * **If**: `count()` **is** `>` **Threshold** `0` |
| 168 | + |
| 169 | +**Notifications**: |
| 170 | + |
| 171 | + * **Choose Notification**: `Report Abuse - HAProxy` |
| 172 | + * **Grace Period**: Disable |
| 173 | + * **Message Backlog**: 500 (duplicates will be filtered by the API-service) |
0 commit comments