Unbound process sporadically returns TOO MANY Servfail and Read/Write errors at different load levels.

[maintain3r]

Unbound version installed: 1.13.1-1ubuntu5.5
unbound runs as a regular service (no as a docker container)
no packet drops are detected on the unbound host
verbosity level is set to 5

The tool to test unbound: dnspyre
The command used to test unbound: dnspyre -c 100 -d 60s --max=20ms -s 172.31.28.217 https://raw.githubusercontent.com/Tantalor93/dnspyre/master/data/10000-domains

Interestingly when I take the domain names that were failing and try to resolve them while the testing tool is not running I do get things resolved properly without an issue.

unbound.conf:
_server:
verbosity: 5
statistics-cumulative: yes
extended-statistics: yes
num-threads: 4
interface: 0.0.0.0
port: 53
prefer-ip6: no
outgoing-range: 8192
outgoing-port-permit: 5354
so-rcvbuf: 8m
so-sndbuf: 8m
so-reuseport: yes
ip-transparent: no
ip-freebind: yes
max-udp-size: 4096
msg-cache-size: 256m
msg-cache-slabs: 8
num-queries-per-thread: 4096
rrset-cache-size: 640m
rrset-cache-slabs: 8
cache-min-ttl: 300
cache-max-ttl: 86400
cache-max-negative-ttl: 300
infra-host-ttl: 60
infra-cache-slabs: 8
infra-cache-numhosts: 100000
do-ip4: yes
do-ip6: no
do-udp: yes
do-tcp: yes
use-systemd: no
do-daemonize: no
access-control: 192.168.0.0/16 allow
access-control: 172.16.0.0/12 allow
access-control: 10.0.0.0/8 allow
access-control: 127.0.0.0/8 allow
username: "unbound"
directory: "/etc/unbound"
use-syslog: no
log-identity: "unbound"
log-time-ascii: yes
log-queries: no
log-replies: yes
log-tag-queryreply: yes
pidfile: "/var/run/unbound.pid"
root-hints: "/var/lib/unbound/root.hints"
hide-identity: yes
hide-version: yes
hide-trustanchor: yes
identity: ""
version: ""
harden-glue: yes
qname-minimisation: yes
use-caps-for-id: yes
do-not-query-localhost: no
prefetch: yes
deny-any: yes
rrset-roundrobin: yes
minimal-responses: yes
val-clean-additional: yes
serve-expired: yes
val-log-level: 2
key-cache-size: 10m
key-cache-slabs: 8
neg-cache-size: 1m
ratelimit: 0
ip-ratelimit: 0

remote-control:
control-enable: yes
control-use-cert: no
control-interface: 127.0.0.1
control-port: 8953
server-key-file: "/etc/unbound/unbound_server.key"
server-cert-file: "/etc/unbound/unbound_server.pem"
control-key-file: "/etc/unbound/unbound_control.key"
control-cert-file: "/etc/unbound/unbound_control.pem"

forward-zone:
name: "."
forward-first: yes
forward-addr: 169.254.169.253@53 # aws provided vpc dns server
forward-addr: 1.1.1.1@53
forward-addr: 8.8.8.8@53_

Testing results
Total requests: 280881
Read/Write errors: 244061
DNS success responses: 34141
DNS negative responses: 1900
DNS error responses: 779

DNS response codes:
NOERROR: 35141
SERVFAIL: 779
NXDOMAIN: 900

DNS question types:
A: 280881

# Running dnspyre locally against 127.0.0.1 (unbound has a listener on this IP). Using 10 concurrent requests didn;t change almost anything, still too many errors.
root@ip-172-31-28-217:/etc/unbound# dnspyre -c 10 -d 60s --max=20ms -s 127.0.0.1 https://raw.githubusercontent.com/Tantalor93/dnspyre/master/data/10000-domains
Using 10000 hostnames
Benchmarking 127.0.0.1:53 via udp with 10 concurrent requests
Total requests: 12844
Read/Write errors: 1134
DNS success responses: 10610
DNS negative responses: 950
DNS error responses: 150

DNS response codes:
NOERROR: 10960
SERVFAIL: 150
NXDOMAIN: 600

DNS question types:
A: 12844

Unbound runs on Ubuntu 22.04.4 LTS
RAM: 4GB
CPU: 2 core
aws t3.medium type host
Changing instance type does not change a lot!!!
CPU usage is ~30-40%

[wcawijngaards]

The setting use-caps-for-id: yes could be the issue, try use-caps-for-id: no. If there is fallback that needs a lot of additional queries, and this option is not common, so I think it causes load and possibly also failures.

With log-servfail: yes it would print out what the servfails are that happen. That would give a clue that point in the direction of what is the cause.

With num-threads: 4, but the host has 2 cpu cores, I would expect num-threads: 2 to be the correct choice. I would not expect that to cause the outcome, but maybe interesting.

The so-rcvbuf and so-sndbuf settings of 8m are large, and I wonder if the 4G host runs out the memory on the many requests that you cause it to queue up for recursion. Out of memory on the socket buffers, and then the recursor cannot make more socket buffers and this causes failure, perhaps.

[maintain3r]

Thanks @wcawijngaards Im gonna try your suggestions and will get back with the results.
For the 'so-rcvbuf' and 'so-sndbuf' what should I use and how to calculate a proper value for that should I create a bigger instance with more RAM ?

[wcawijngaards]

I do not know a value calculation for them. Perhaps leave them at default. Or 64k for less buffer size but also less memory consumption, since the test involves opening thousands of sockets.

[maintain3r]

Taken from unbound official doc page:
Set so-rcvbuf to a larger value (4m or 8m) for a busy server. This sets the kernel buffer larger so that no messages are lost in spikes in the traffic. Adds extra 9s to the reply-reliability percentage. The OS caps it at a maximum, on linux unbound needs root permission to bypass the limit, or the admin can use sysctl net.core.rmem_max. On BSD change kern.ipc.maxsockbuf in /etc/sysctl.conf.