feat: add `on_skipped_request` decorator, to process links skipped according to `robots.txt` rules#1166

Merged

Pijukatel merged 37 commits intoapify:masterfrom

Mantisus:on-skipped-request

May 2, 2025

Commits on Apr 21, 2025

Commits on Apr 24, 2025

change staticmethod to classmethod
Mantisus
committed
Update src/crawlee/_utils/robots.py

Mantisus
and
vdusek
committed
add _robots_txt_locks_cache
Mantisus
committed
update `pyproject.toml`
Mantisus
committed
update docstrings
Mantisus
committed
add docs example
Mantisus
committed
chore(deps): update typescript-eslint monorepo to v8.31.0 (#1168)

renovate[bot]
authored and
Mantisus
committed
chore(deps): update dependency setuptools to v79 (#1165)

renovate[bot]
authored and
Mantisus
committed
fix: Update `UnprocessedRequest` to match actual data (#1155)

Pijukatel
authored and
Mantisus
committed
chore(release): Update changelog and package version [skip ci]

Apify Release Bot
authored and
Mantisus
committed
fix: Fix the order in which cookies are saved to the `SessionCookies` and the handler is executed for `PlaywrightCrawler` (#1163)
Mantisus
committed
chore(release): Update changelog and package version [skip ci]

Apify Release Bot
authored and
Mantisus
committed
feat: Handle unprocessed requests in `add_requests_batched` (#1159)

Pijukatel
authored and
Mantisus
committed
chore(release): Update changelog and package version [skip ci]

Apify Release Bot
authored and
Mantisus
committed
fix: call `failed_request_handler` for `SessionError` when session rotation count exceeds maximum (#1147)
Mantisus
committed
chore(release): Update changelog and package version [skip ci]

Apify Release Bot
authored and
Mantisus
committed
one lock to rule them all
Mantisus
committed
Merge branch 'master' into on-skipped-request
Mantisus
committed
fix
Mantisus
committed

Commits on Apr 25, 2025

add docs
Mantisus
committed

Commits on Apr 28, 2025