fix: throttle Azure OpenAI embedding calls and back off on rate limits - #49
Open
janton10 wants to merge 1 commit into
Open
fix: throttle Azure OpenAI embedding calls and back off on rate limits#49janton10 wants to merge 1 commit into
janton10 wants to merge 1 commit into
Conversation
The data loader previously fanned out all 4 contribution types
("Spoken", "Written", "Corrections", "Petitions") concurrently in a
TaskGroup, with each type spawning one task per page. With nothing
limiting concurrent embed_batch calls, the loader could trivially
exceed Azure's TPM quota at lower tiers. The retry decorator
stop_after_attempt(3) had no wait, so it ignored the Retry-After header.
Changes:
- Add an aiolimiter + semaphore around every embeddings.create call,
configurable via OPENAI_EMBED_RATE_PER_SECOND, OPENAI_EMBED_MAX_CONCURRENCY,
OPENAI_EMBED_BATCH_SIZE env vars.
- Switch tenacity to wait_random_exponential with jitter, retry only on
RateLimitError / APITimeoutError / APIConnectionError, up to 8 attempts.
- Pass max_retries=6 to AsyncAzureOpenAI so the SDK honours Retry-After
before tenacity takes over for longer waits.
- Reduce default batch_size from 100 to 32 (~10k tokens/call) to fit
the per-minute quota.
- Run load_all_contributions sequentially; pages within each type still
parallelise.
janton10
force-pushed
the
fix/embed-rate-limit
branch
from
April 25, 2026 17:35
e92ced3 to
5078f8b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
make load-data hansardreliably 429s on the default Azure S0 tier.load_all_contributionsfans out 4 contribution types in parallel x N pages each,batch_size=100puts ~30k tokens per call (well over 350k TPM in bursts), and@retry(stop_after_attempt(3))retries instantly with no backoff and ignoresRetry-After.Patch:
embeddings.createwith anaiolimiter+ semaphore. Tune viaOPENAI_EMBED_RATE_PER_SECOND/OPENAI_EMBED_MAX_CONCURRENCY/OPENAI_EMBED_BATCH_SIZE. Defaults sized for S0; raise on larger quotas.wait_random_exponential(2, 60), retries only onRateLimitError/APITimeoutError/APIConnectionError, up to 8 attempts.max_retries=6on the Azure client so the SDK honoursRetry-Afterfirst.batch_size100 to 32.load_all_contributionsruns types sequentially; pages still parallelise within each type.No new deps. 90-day Hansard load completes on S0 with these defaults.