WPB-11455 Allow List for 401 rate limiting #24

e-lisa · 2024-11-12T00:07:54Z

PR Submission Checklist for internal contributors

The PR Title
- conforms to the style of semantic commits messages¹ supported in Wire's Github Workflow²
- contains a reference JIRA issue number like SQPIT-764
- answers the question: If merged, this PR will: ... ³
The PR Description
- is free of optional paragraphs and you have filled the relevant parts to the best of your ability

What's new in this PR?

Allowlist for the 401 rate-limit features.

Added a new --401-allowlist to allow adding a path of an allow-list of
IPv4/IPv6 addresses that can bypass the 401 rate-limit settings. This
can also be set in the configuration file via 401-allowlist=.

The allow-list is updated when the allowlist file is updated during
runtime.

The allow-list format is one (1) IP address per line.

Issues

An allow-list to bypass the 401 rate-limit feature is required for federated services and other high volume IP addresses.

Solutions

A new option --401-allowlist= and configuration file setting 401-allowlist= have been added to allow specifying a file to contain the allow list. The allow list is updated every time the allow list file is updated based on its mtime (modification time).

The allow list is formatted with one IPv4/IPv6 address per line as seen in this example allow list:

127.0.0.1
192.168.1.5

Needs releases with:

GitHub link to other pull request

Testing

Test Coverage (Optional)

I have added automated test to this contribution

How to Test

Briefly describe how this change was tested and if applicable the exact steps taken to verify that it works as expected.

Notes (Optional)

Specify here any other facts that you think are important for this issue.

Attachments (Optional)

Attachments like images, videos, etc. (drag and drop in the text box)

PR Post Submission Checklist for internal contributors (Optional)

Wire's Github Workflow has automatically linked the PR to a JIRA issue

PR Post Merge Checklist for internal contributors

If any soft of configuration variable was introduced by this PR, it has been added to the relevant documents and the CI jobs have been updated.

References

https://sparkbox.com/foundry/semantic_commit_messages
https://github.com/wireapp/.github#usage
E.g. feat(conversation-list): Sort conversations by most emojis in the title #SQPIT-764.

lwille · 2024-11-14T17:02:06Z

src/server/ns_turn_ratelimit.c

+
+  /* Check the mtime of the allow list, do we need to update? */
+  struct stat fstat;
+  if (stat(allowlist, &fstat) == 0) {


question: are we opening, reading and closing the allowlist file for each request?

We are doing stat() and then opening the file if and only if the mtime changes between rate-limited 401 requests

We could try setting a "timeout" for how often the allow-list is updated, but then we'd still take a small hit for comparing the times, in addition to checking the mtime and opening the file. Both way seem valid. But to me this seems like less bloat.

Was there a requirement that the allow list be editable without restarting? If not, I suppose another option is to just put the entire allow list in the configuration file and load it one at startup.

@sgodin so @julialongtin told me that it needs to be updated during runtime, otherwise yes we can move this to the server initialization code. I suspect we want it to update during runtime to prevent having to disruptions during adding/removing federated servers and adding larger users to the allow list.

Yes, this is an operational requirement. A restart of this service is a Big Deal(TM).

Thank you for elaborating, that sounds very reasonable.

I think a stat does a system call which reads inode info, I think we should avoid doing this.
Better to update the allowlist on a signal?

signal would safer and avoid potential blocking from filesystem IO :) There is a signal handler already that re-reads certificate files, perhaps you can attach to that.

Let me look to see how viable this is. Mind you this wouldn't be every request, as this is only run when we do the rate-limiting - not every request

But still, I will try to figure out ASAP if its possible to attach this to a signal to reload the file

src/server/ns_turn_ratelimit.c

src/ns_turn_defs.h

sgodin · 2024-11-14T17:46:56Z

It would be good to add all your new command line / configuration options to the sample turnserver.conf file:
https://github.com/wireapp/coturn/blob/master/examples/etc/turnserver.conf

Added a new `--401-allowlist` to allow adding a path of an allow-list of IPv4/IPv6 addresses that can bypass the 401 rate-limit settings. This can also be set in the configuration file via `401-allowlist=.` The allow-list is updated when the allowlist file is updated during runtime. The allow-list format is one (1) IP address per line.

e-lisa · 2024-11-18T10:48:10Z

It would be good to add all your new command line / configuration options to the sample turnserver.conf file: https://github.com/wireapp/coturn/blob/master/examples/etc/turnserver.conf

added

sgodin · 2024-11-20T20:47:21Z

@e-lisa If you can please avoid squashing the commits (at least for now), it will be easier to see what has been changed from the commits that result from review comments. :)

e-lisa · 2024-12-16T12:13:23Z

@sgodin Added a global lock for the ur_addr_map_get() method. Let me know if you think there are more changes required.

src/server/ns_turn_ratelimit.c

z-dule

I made a few inline comments.

src/server/ns_turn_ratelimit.c

…IGRTMIN+3

…into WPB-11455

e-lisa · 2024-12-26T11:34:57Z

Moved all the allowlist loading and initialization code to the mainserver startup in mainrelay.c

Assigned the real-time signal SIGRTMIN+3 to reloading the allowlist.

SIGUSR1 and SIGUSR2 are already used for single task jobs (drain and loading the TLS certificates), SIGHUP is used for reloading the configuration files (we could hook into this if wanted). SIGRTMIN+3 seemed like the best option with out changing the functionality or reloading the configuration files.

…e() in the future

sgodin · 2025-01-03T16:22:32Z

src/server/ns_turn_ratelimit.c

-        char line[1024];
+    /* Rebuild map */
+    TURN_MUTEX_LOCK(&rate_limit_allowlist_mutex);
+    ur_addr_map_clean(rate_limit_allowlist_map);


You probably want to hold the lock for the entire read process, otherwise you run the risk of rate limiting an allowed endpoint between the list being cleared and the entries being added back.

z-dule

LGTM not much more to add.

mastaab · 2025-01-17T08:37:07Z

@z-dule can you please go ahead and merge it?

echoes-hq bot added the echoes/initiative: federation-and-mls-on-wire-c... Activate Federation with MLS on Wire Cloud label Nov 12, 2024

e-lisa force-pushed the WPB-11455 branch from 509835f to 9d52c19 Compare November 12, 2024 12:06

e-lisa marked this pull request as ready for review November 12, 2024 12:06

jschaul requested a review from supersven November 14, 2024 16:09

lwille reviewed Nov 14, 2024

View reviewed changes

sgodin reviewed Nov 14, 2024

View reviewed changes

src/server/ns_turn_ratelimit.c Show resolved Hide resolved

sgodin reviewed Nov 14, 2024

View reviewed changes

src/server/ns_turn_ratelimit.c Outdated Show resolved Hide resolved

sgodin reviewed Nov 14, 2024

View reviewed changes

src/server/ns_turn_ratelimit.c Show resolved Hide resolved

sgodin reviewed Nov 14, 2024

View reviewed changes

src/ns_turn_defs.h Outdated Show resolved Hide resolved

e-lisa force-pushed the WPB-11455 branch from 9d52c19 to 703e225 Compare November 15, 2024 10:53

e-lisa force-pushed the WPB-11455 branch from 703e225 to f48a2eb Compare November 18, 2024 10:00

e-lisa requested review from sgodin and lwille November 18, 2024 10:02

Added global lock for addr_map_get, for threading

c7d952b

Bumped coturn version

d1fb9cc

z-dule reviewed Dec 16, 2024

View reviewed changes

src/server/ns_turn_ratelimit.c Show resolved Hide resolved

z-dule reviewed Dec 16, 2024

View reviewed changes

src/server/ns_turn_ratelimit.c Outdated Show resolved Hide resolved

z-dule reviewed Dec 16, 2024

View reviewed changes

sgodin reviewed Dec 19, 2024

View reviewed changes

src/server/ns_turn_ratelimit.c Outdated Show resolved Hide resolved

sgodin reviewed Dec 19, 2024

View reviewed changes

src/server/ns_turn_ratelimit.c Show resolved Hide resolved

e-lisa added 2 commits December 26, 2024 12:19

Moved allowlist init to main init, moved allowlist reload to signal S…

2812782

…IGRTMIN+3

Merge branch 'wireapp/4.6.2-federation' of github.com:wireapp/coturn …

b4bcbcf

…into WPB-11455

e-lisa added 2 commits December 26, 2024 12:53

cleaned up ratelimit logic with a goto/label to help avoid double fre…

8ec2e46

…e() in the future

cleaned up ratelimit logic with a goto/label to help avoid double fre…

78cef27

…e() in the future

e-lisa requested review from z-dule, sgodin and typfel December 26, 2024 11:56

sgodin reviewed Jan 3, 2025

View reviewed changes

z-dule reviewed Jan 13, 2025

View reviewed changes

z-dule merged commit 0c959ed into wireapp:wireapp/4.6.2-federation Jan 31, 2025
2 checks passed

mastaab mentioned this pull request Feb 10, 2025

add helm config for the allowlist of binding requests rate-limiting #31

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WPB-11455 Allow List for 401 rate limiting #24

WPB-11455 Allow List for 401 rate limiting #24

e-lisa commented Nov 12, 2024 •

edited

Loading

lwille Nov 14, 2024

e-lisa Nov 14, 2024 •

edited

Loading

sgodin Nov 14, 2024

e-lisa Nov 14, 2024 •

edited

Loading

julialongtin Nov 14, 2024

lwille Nov 15, 2024

z-dule Dec 16, 2024 •

edited

Loading

sgodin Dec 19, 2024

e-lisa Dec 20, 2024

sgodin commented Nov 14, 2024

e-lisa commented Nov 18, 2024

sgodin commented Nov 20, 2024

e-lisa commented Dec 16, 2024

z-dule left a comment

e-lisa commented Dec 26, 2024

sgodin Jan 3, 2025

z-dule left a comment

mastaab commented Jan 17, 2025

WPB-11455 Allow List for 401 rate limiting #24

WPB-11455 Allow List for 401 rate limiting #24

Conversation

e-lisa commented Nov 12, 2024 • edited Loading

PR Submission Checklist for internal contributors

What's new in this PR?

Issues

Solutions

Testing

Test Coverage (Optional)

How to Test

Notes (Optional)

Attachments (Optional)

PR Post Submission Checklist for internal contributors (Optional)

PR Post Merge Checklist for internal contributors

References

lwille Nov 14, 2024

Choose a reason for hiding this comment

e-lisa Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

sgodin Nov 14, 2024

Choose a reason for hiding this comment

e-lisa Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

julialongtin Nov 14, 2024

Choose a reason for hiding this comment

lwille Nov 15, 2024

Choose a reason for hiding this comment

z-dule Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

sgodin Dec 19, 2024

Choose a reason for hiding this comment

e-lisa Dec 20, 2024

Choose a reason for hiding this comment

sgodin commented Nov 14, 2024

e-lisa commented Nov 18, 2024

sgodin commented Nov 20, 2024

e-lisa commented Dec 16, 2024

z-dule left a comment

Choose a reason for hiding this comment

e-lisa commented Dec 26, 2024

sgodin Jan 3, 2025

Choose a reason for hiding this comment

z-dule left a comment

Choose a reason for hiding this comment

mastaab commented Jan 17, 2025

e-lisa commented Nov 12, 2024 •

edited

Loading

e-lisa Nov 14, 2024 •

edited

Loading

e-lisa Nov 14, 2024 •

edited

Loading

z-dule Dec 16, 2024 •

edited

Loading