[AAP-82119] Batch resource migration using bulk-update endpoint - #167
Conversation
|
Warning Review limit reached
Next review available in: 17 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughResource migration now processes upstream resources in batches, creates local Gateway resources transactionally, retries transient bulk-update failures, and tracks progress to detect stalled migrations. Tests cover payloads, retries, partial updates, and circuit breaking. ChangesResource migration batching and contracts
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ResourceMigration
participant GatewayDatabase
participant GWResourceAPIClient
participant UpstreamService
ResourceMigration->>GatewayDatabase: Create local resources for page
ResourceMigration->>GWResourceAPIClient: Send chunked bulk update items
GWResourceAPIClient->>UpstreamService: POST resources/bulk-update/
UpstreamService-->>GWResourceAPIClient: Return update results or transient failure
GWResourceAPIClient-->>ResourceMigration: Return successful update count
ResourceMigration->>ResourceMigration: Detect zero-progress pages
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dostonbek1
changed the title
1b15bba to
4260a1d
Compare
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## devel #167 +/- ##
==========================================
+ Coverage 94.39% 94.40% +0.01%
==========================================
Files 138 138
Lines 6404 6487 +83
Branches 792 812 +20
==========================================
+ Hits 6045 6124 +79
- Misses 265 267 +2
- Partials 94 96 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
8f04a0d to
26f6673
Compare
✅ Test Results - PASSEDSummary
Pass Rate: 98.2% |
github-actions
Bot
added
fix-ci
26f6673 to
42972a0
Compare
github-actions
Bot
removed
fix-ci
1ea1d9c to
1768848
Compare
✅ Test Results - PASSEDSummary
Pass Rate: 97.0% |
github-actions
Bot
added
fix-ci
Testing Results: Production-Scale End-to-End Migration BenchmarkTested against a local AAP dev instance (AAP 2.7, controller 4.8.3) with production-scale data:
Performance Results (Controller Service-Index Endpoint)Benchmark simulates the exact path
Comparison with Role Assignment Optimization (AAP-80609)
Platform Services Test SuiteRan The 2 failures are pre-existing and unrelated (service_cluster PATCH proxy permissions). Acceptance Criteria
Result: 9.2 seconds — well under target. ✅ The full |
Dostonbek1
changed the title
github-actions
Bot
added
fix-ci
717e871 to
d59dfe9
Compare
github-actions
Bot
removed
fix-ci
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
aap_gateway_api/utils/resources_client.py (1)
39-43: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocstring lists
service_id, but the codebase usesnew_service_id.The docstring says items may contain
service_id, but_build_bulk_update_item(and its tests) build/expectnew_service_id. Update the docstring to match the actual contract.Proposed docstring fix
Each item must contain 'ansible_id' and one or more fields to update: - service_id, new_ansible_id, is_partially_migrated, resource_data. + new_service_id, new_ansible_id, is_partially_migrated, resource_data.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@aap_gateway_api/utils/resources_client.py` around lines 39 - 43, Update the bulk-update method’s docstring to list new_service_id instead of service_id, matching the field constructed and expected by _build_bulk_update_item and its tests; leave the other documented fields unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@aap_gateway_api/utils/resources_client.py`:
- Around line 39-43: Update the bulk-update method’s docstring to list
new_service_id instead of service_id, matching the field constructed and
expected by _build_bulk_update_item and its tests; leave the other documented
fields unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Enterprise
Run ID: 72175620-905e-461a-9c5c-22c577e840ca
📒 Files selected for processing (3)
aap_gateway_api/management/commands/_migrate_service_data/resource_migration.pyaap_gateway_api/tests/management/commands/_migrate_service_data/test_resource_migration.pyaap_gateway_api/utils/resources_client.py
🚧 Files skipped from review as they are similar to previous changes (1)
- aap_gateway_api/management/commands/_migrate_service_data/resource_migration.py
798bf03 to
a23bdb3
Compare
github-actions
Bot
added
Review-provided
✅ Test Results - PASSEDSummary
Pass Rate: 97.0% |
github-actions
Bot
added
fix-ci
Refactor migrate_resource() to process entire API pages as batches instead of one item at a time. The new _process_resource_page_batch() method: 1. Validates all items on the page (reuses existing helpers) 2. Reconciles against existing gateway resources 3. Bulk-creates gateway resources in a single transaction 4. Sends a single bulk_update_resources() HTTP call per page Key improvements: - Chunk bulk_update_items to MAX_BULK_CHUNK_SIZE (1000) to respect upstream endpoint limits - Add retry with exponential backoff for transient errors (502/503/504, network errors) — up to 3 attempts before giving up - Fail immediately on permanent errors (4xx) without retry - Treat non-JSON responses as transient (likely proxy errors) and retry - Circuit breaker: abort after 3 consecutive pages with zero progress - Align with DAB API rename of service_id → new_service_id - Fix UUID serialization in bulk-update payload (str() conversion) This reduces HTTP round-trips from N (one per resource) to N/page_size (one per page). For Kyndryl-scale data (9,881 resources), this cuts total migration time from 11m38s to 2m40s — a 4.4x speedup / 77% time reduction. Depends on: django-ansible-base AAP-82119/bulk-update-endpoint branch Assisted-by: Claude AI assistant
a23bdb3 to
8fd8773
Compare
github-actions
Bot
removed
fix-ci
|
DVCS PR Check Results: PR appears valid (JIRA key(s) found) |
|
✅ Test Results - PASSEDSummary
Pass Rate: 97.0% |
github-actions
Bot
added
the
ready-to-merge
github-actions
Bot
removed
Review-provided
4 participants
Requires: ansible/django-ansible-base#1054
Description
migrate_resource()to process entire API pages as batches instead of one item at a time, using the new DAB bulk-update endpoint._process_resource_page_batch()method validates all items on the page, reconciles conflicts, bulk-creates gateway resources in a single transaction, then sends onebulk_update_resources()HTTP call per page.Type of Change
Self-Review Checklist
Testing Instructions
Prerequisites
Steps to Test
manage.py migrate_service_data --username adminExpected Results
Additional Context
Required Actions
Performance Impact
Note: This PR was developed with assistance from Claude AI assistant.
Summary by CodeRabbit
New Features
Bug Fixes
new_service_idfield name.Tests