Recommended way to poke rtd.org programmatically

[flying-sheep]

Details

We have CI that checks a list of links for availability. It stopped working when you made a change designed to block AI crawlers

See scverse/scverse-tutorials#236 (comment)

Expected Result

There should be guidance how to make automated requests to rtd.org without being classified as AI crawler.

Actual Result

403 even when doing HEAD requests and setting a custom user agent.

Using browser automation to do the request works, but we want to be good netizens and do things by the book instead of pretending we’re a browser.

[ericholscher]

What is the user agent you're using? I can check the logs and see what is blocking things.

[flying-sheep]

I just tried with "scverse tutorial registry (https://github.com/scverse/scverse-tutorials)"

It raised a 403 for https://muon-tutorials.readthedocs.io/en/latest/cite-seq/1-CITE-seq-PBMC-5k.html

See: https://github.com/scverse/scverse-tutorials/actions/runs/17675527461/job/50236608525?pr=236

Thanks!

[ericholscher]

This is getting blocked because the traffic (JA4) looks the same as a large DDOS that is attacking us from many different countries. Unfortunately there isn't really much I can do at this point, but I would suggest trying to use another client in some fashion. I thought the JA4 was more unique and unlikely to cause issues like this, but we will try and see what we can do.

[flying-sheep]

I’m sorry to hear that you’re being DDOS’d, but I’m confused.

Our requests come from GitHub IPs, have a totally custom user agent, and are only HEAD requests. Surely that shouldn’t match DDOS traffic … idk what we can do other than going back to using playwright and pretending we’re a browser.

[flying-sheep]

Can you just allowlist HEAD requests? They are probably cheap to answer and all we need.

[ericholscher]

I've removed HEAD requests from the block.

[flying-sheep]

Thank you! But it looks like that only worked for some URLs, now it fails trying to reach https://scanpy.readthedocs.io/en/stable/tutorials/plotting/advanced.html:

https://github.com/scverse/scverse-tutorials/actions/runs/17735794561/job/50397181823?pr=236

I assume they’re tried in order, which would mean that the links with index 0–3 succeed in the following list.

I guess we could change everything to use the sphinx inventory if that’s unblocked everywhere, but that’s more strain on your servers than HEAD requests …

╭────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│  0 │ https://muon-tutorials.readthedocs.io/en/latest/single-cell-rna-atac/pbmc10k/2-Chromatin-Accessibility-Processing.html            │
│  1 │ https://muon-tutorials.readthedocs.io/en/latest/cite-seq/1-CITE-seq-PBMC-5k.html                                                  │
│  2 │ https://docs.scvi-tools.org/en/stable/tutorials/notebooks/multimodal/MultiVI_tutorial.html                                        │
│  3 │ https://scvelo.readthedocs.io/en/stable/getting_started.html                                                                      │
│  4 │ https://scanpy.readthedocs.io/en/stable/tutorials/plotting/advanced.html                                                          │
│  5 │ https://scverse-tutorials.readthedocs.io/en/latest/notebooks/anndata_getting_started.html                                         │
│  6 │ https://scverse-tutorials.readthedocs.io/en/latest/notebooks/tutorial_axes_anndata_mudata.html                                    │
│  7 │ https://squidpy.readthedocs.io/en/latest/notebooks/tutorials/tutorial_vizgen_mouse_liver.html                                     │
│  8 │ https://www.sc-best-practices.org/conditions/compositional.html                                                                   │
│  9 │ https://scverse-tutorials.readthedocs.io/en/latest/notebooks/tutorial_concatenation_anndata_mudata.html                           │
│ 10 │ https://anndata.readthedocs.io/en/latest/concatenation.html                                                                       │
│ 11 │ https://decoupler.readthedocs.io/en/latest/notebooks/scell/rna_psbk.html                                                          │
│ 12 │ https://scverse-tutorials.readthedocs.io/en/latest/notebooks/scverse_data_interoperability.html                                   │
│ 13 │ https://www.sc-best-practices.org/conditions/perturbation_modeling.html                                                           │
│ 14 │ https://scanpy.readthedocs.io/en/stable/tutorials/plotting/core.html                                                              │
│ 15 │ https://scverse-tutorials.readthedocs.io/en/latest/notebooks/basic-scrna-tutorial.html                                            │
│ 16 │ https://www.sc-best-practices.org/trajectories/pseudotemporal.html#                                                               │
│ 17 │ https://scirpy.scverse.org/en/latest/tutorials/tutorial_3k_tcr.html                                                               │
│ 18 │ https://scverse-tutorials.readthedocs.io/en/latest/notebooks/scverse_data_backed.html#working-with-scverse-objects-in-backed-mode │
│ 19 │ https://docs.scvi-tools.org/en/stable/tutorials/notebooks/quick_start/api_overview.html                                           │
│ 20 │ https://bento-tools.readthedocs.io/en/latest/index.html                                                                           │
│ 21 │ https://cellcharter.readthedocs.io/en/latest/notebooks/cosmx_human_nsclc.html                                                     │
╰────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[ericholscher]

I don't see any requests from that user agent getting blocked by our WAF, so it must be getting blocked by something else. We haven't changed our firewall settings except for this specific JA4 block, which is still sending 1M+ requests a day. We can look at trying to allow list GH IPs, but that is a bit larger of a project.

[flying-sheep]

Maybe an older change. That CI stopped working between Jan 27 and Feb 4 this year.