[hma] Support fetching media from S3-compatible storage - #1997
Closed
julietshen wants to merge 1 commit into
Closed
Conversation
Teach the shared URL-fetch path to understand s3://bucket/key URLs so any endpoint that already accepts a `url` (GET /h/hash, banking content by URL, matching by URL, the UI) can hash media stored in S3-compatible object storage (AWS S3, MinIO, DigitalOcean Spaces, etc) without a caller-side download step. - hash_url_content() branches on the s3 scheme; the http(s) path is unchanged. - New hash_s3_content(): head_object for content-type + size pre-check, then streams get_object into a temp file, re-enforcing the size cap mid-stream in case ContentLength is absent or wrong. - Extract _resolve_max_remote_file_size() shared by both fetch paths. - boto3 is an optional extra (OpenMediaMatch[s3]); a guarded import returns a clear 500 if it is missing. - Config: S3_ENDPOINT_URL / S3_REGION_NAME for S3-compatible stores, and an optional ALLOWED_S3_BUCKETS allowlist mirroring ALLOWED_HOSTNAMES (S3 URLs intentionally bypass the http SSRF/hostname checks, so this is the parallel guardrail). - Tests cover happy path, endpoint/region wiring, invalid URLs, allowlist allow/deny, oversize by header and mid-stream, 404, and missing-boto3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jqavcfws33XLnHAfwoXXun
github-actions
Bot
added
the
hma
reitblatt
added a commit
to reitblatt/ThreatExchange
that referenced
this pull request
Sep 9, 2026
Extend hash_url_content() to recognise the s3:// scheme so every endpoint that already accepts a `url` (GET /h/hash, banking content by URL, matching by URL, the UI) gains S3-compatible object storage support -- no new endpoint, no request-body changes. - hash_s3_content(): head_object pre-check for content type and size, then streams get_object into a temp file re-enforcing the size cap mid-stream (in case ContentLength is absent or wrong). - _parse_s3_url() splits s3://bucket/key (400 on a malformed URL); _get_s3_client() builds the boto3 client from S3_ENDPOINT_URL / S3_REGION_NAME with credentials from boto3's default provider chain. - boto3 is an optional dependency (OpenMediaMatch[s3]); a guarded import returns a clear 500 when it is missing, keeping the base image lean. - s3:// URLs intentionally bypass the http SSRF / ALLOWED_HOSTNAMES checks (they are authenticated bucket reads); ALLOWED_S3_BUCKETS is the parallel allowlist guardrail. - MAX_REMOTE_FILE_SIZE handling extracted into _resolve_max_remote_file_size() and shared by the http and s3 paths. Design follows facebook#1997. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CNUbBXRQzLeMHn5exLBHS5
reitblatt
added a commit
to reitblatt/ThreatExchange
that referenced
this pull request
Sep 10, 2026
Give hash_url_content() an s3:// branch so every endpoint that already
accepts a `url` (GET /h/hash, banking content by URL, matching by URL, the
UI) can hash objects in S3-compatible storage -- no new endpoint, no
request-body changes.
The http and s3 paths now share their common machinery instead of carrying
near-identical copies:
* _resolve_max_remote_file_size() - MAX_REMOTE_FILE_SIZE coercion
* _resolve_signal_types() - source content-type + overrides -> signal types
* _hash_chunks_to_signals() - spool a byte stream to a temp file
(re-checking the size cap mid-stream) and
run every FileHasher over it
S3 specifics:
* _s3_split_url() - s3://bucket/key -> (bucket, key), 400 on a bad URL, and
enforces the optional ALLOWED_S3_BUCKETS allowlist (the
S3 analogue of ALLOWED_HOSTNAMES; s3:// deliberately
bypasses the http SSRF checks as it is an authenticated
bucket read)
* _get_s3_client() - lazy/guarded boto3 import (500 with an install hint when
the optional 's3' extra is absent); honours
S3_ENDPOINT_URL / S3_REGION_NAME
* _s3_call() - one boto3 call wrapper mapping botocore failures to HTTP
(403/404 through, other API errors 400, transport 502)
without echoing the provider message
boto3 is an optional dependency (OpenMediaMatch[s3]). Config keys and the
schema `url` descriptions (HashRequest, matching.LookupRequest,
curation.BankContentRequest, ui.QueryUrlRequest, ui.BankFindContentRequest)
document the s3:// support.
Docs/config adapted from facebook#1997.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CNUbBXRQzLeMHn5exLBHS5
reitblatt
added a commit
to reitblatt/ThreatExchange
that referenced
this pull request
Sep 10, 2026
Give hash_url_content() an s3:// branch so every endpoint that already
accepts a `url` (GET /h/hash, banking content by URL, matching by URL, the
UI) can hash objects in S3-compatible storage -- no new endpoint, no
request-body changes.
The http and s3 paths now share their common machinery instead of carrying
near-identical copies:
* _resolve_max_remote_file_size() - MAX_REMOTE_FILE_SIZE coercion
* _resolve_signal_types() - source content-type + overrides -> signal types
* _hash_chunks_to_signals() - spool a byte stream to a temp file
(re-checking the size cap mid-stream) and
run every FileHasher over it
S3 specifics:
* _s3_split_url() - s3://bucket/key -> (bucket, key), 400 on a bad URL, and
enforces the optional ALLOWED_S3_BUCKETS allowlist (the
S3 analogue of ALLOWED_HOSTNAMES; s3:// deliberately
bypasses the http SSRF checks as it is an authenticated
bucket read)
* _get_s3_client() - lazy/guarded boto3 import (500 with an install hint when
the optional 's3' extra is absent); honours
S3_ENDPOINT_URL / S3_REGION_NAME
* _s3_call() - one boto3 call wrapper mapping botocore failures to HTTP
(403/404 through, other API errors 400, transport 502)
without echoing the provider message
boto3 is an optional dependency (OpenMediaMatch[s3]). Config keys and the
schema `url` descriptions (HashRequest, matching.LookupRequest,
curation.BankContentRequest, ui.QueryUrlRequest, ui.BankFindContentRequest)
document the s3:// support.
Docs/config adapted from facebook#1997.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CNUbBXRQzLeMHn5exLBHS5
Contributor
Author
|
Closing in lieu of #1998 ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1828.
Why
Today, to hash media in S3-compatible storage you must either presign a URL or run an intermediary that downloads the object and re-uploads it to HMA. This adds
s3://bucket/keysupport so HMA can fetch directly.What
Rather than a new endpoint,
hash_url_content()now branches on thes3URL scheme, so every endpoint that already accepts aurl(GET /h/hash, banking content by URL, matching by URL, the UI) gains S3 support. Thehttp(s)path is unchanged.hash_s3_content():head_objectfor content-type + size pre-check, then streamsget_objectinto a temp file, re-enforcing the size cap mid-stream in caseContentLengthis absent/wrong.OpenMediaMatch[s3]); a guarded import returns a clear 500 if missing, so the base image stays lean.S3_ENDPOINT_URL/S3_REGION_NAMEfor S3-compatible stores (MinIO, DigitalOcean Spaces), plus an optionalALLOWED_S3_BUCKETSallowlist. Note: S3 URLs intentionally bypass thehttpSSRF /ALLOWED_HOSTNAMESchecks (they are authenticated bucket reads), soALLOWED_S3_BUCKETSis the parallel guardrail — flagging this for reviewer scrutiny.Test plan
Ran the CI jobs locally in Docker against a Postgres container:
black --check— cleanmypy—Success: no issues found in 50 source filespytest test_hashing.py— 20 passed (9 existing + 11 new S3 cases: happy path, endpoint/region wiring, invalid URLs, allowlist allow/deny, oversize-by-header, oversize-mid-stream, 404, missing-boto3)AI assistance disclosure
This change was developed with AI assistance (Claude). Per the repo's AI Coding Policy, the design decisions and code have been reviewed and the author can vouch for them; the one area to point reviewers to for extra scrutiny is the SSRF-bypass tradeoff noted above.
Opened as a draft to serve as a reference implementation for a contributor picking up #1828.