동적 워크플로우 기반 배치 시스템 초기 설정 #26
Conversation
- ProductCrawlingData: 배치 작업의 Reader -> Processor -> Writer 단계 사이에서 데이터를 실어 나를 내부용 데이터 상자 - FastApiCrawlingResponse: RestTemplate이 FastAPI 서버에 크롤링을 요청한 후, 그 응답(JSON)을 담아올 외부용 데이터 상자
- ProductCrawlingData라는 상자에 DB 데이터를 담거나, 상자의 내용물로 DB를 업데이트할 수 있도록 실제 SQL 쿼리를 CrawlingMapper.xml에 작성하고, 이를 Java에서 호출할 수 있도록 CrawlingMapper.java 인터페이스로 연결
There was a problem hiding this comment.
Pull Request Overview
This PR implements a batch processing system for product crawling using Spring Batch. The feature enables scheduled crawling of product information from external APIs to update product data (price and stock status) in the database.
Key changes:
- Adds Spring Batch infrastructure for product crawling with parallel processing and fault tolerance
- Implements a scheduled task that runs daily at 8 AM to trigger the batch job
- Creates DTOs and MyBatis mappers for product data handling
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| build.gradle | Adds Spring Batch starter dependency |
| UserServiceApplication.java | Enables scheduling support |
| CrawlingBatchConfig.java | Configures batch job with reader, processor, writer and fault tolerance |
| CrawlingScheduler.java | Implements scheduled task to trigger crawling job |
| CrawlingMapper.java | Defines MyBatis mapper interface for database operations |
| CrawlingMapper.xml | Contains SQL queries for reading and updating product data |
| ProductCrawlingData.java | DTO for product data transfer |
| FastApiCrawlingResponse.java | DTO for external API response |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| id | ||
| LIMIT #{_pagesize} OFFSET #{_skiprows} |
There was a problem hiding this comment.
The parameter names '_pagesize' and '_skiprows' use underscore-prefixed naming which is inconsistent with MyBatis conventions. These should be changed to 'pageSize' and 'skipRows' or the mapper method should use @param annotations to properly map these parameters.
| id | |
| LIMIT #{_pagesize} OFFSET #{_skiprows} | |
| LIMIT #{pageSize} OFFSET #{skipRows} |
| log.info("Requesting crawl for product: {}", product.getName()); | ||
|
|
||
| // FastAPI URL을 코드에 직접 작성 (추후 외부 설정으로 분리 권장) | ||
| String fastApiUrl = "http://your-fastapi-server.com/crawl?url=" + product.getUrlToCrawl(); |
There was a problem hiding this comment.
The FastAPI URL is hardcoded and uses a placeholder hostname. This should be externalized to application properties for different environments and proper configuration management.
There was a problem hiding this comment.
하드 코딩 되어있는 Fast API의 URL은 추후에 수정할 예정
| log.info("Requesting crawl for product: {}", product.getName()); | ||
|
|
||
| // FastAPI URL을 코드에 직접 작성 (추후 외부 설정으로 분리 권장) | ||
| String fastApiUrl = "http://your-fastapi-server.com/crawl?url=" + product.getUrlToCrawl(); |
There was a problem hiding this comment.
URL concatenation without proper encoding could lead to URL injection vulnerabilities. Use UriComponentsBuilder or similar to safely construct URLs with parameters.
|
@jihukimme
실행 로직 관리(재처리, 상태관리)가 필요하기 때문에 batch가 좋을 것 같아요 |
|
@jihukimme 완료되면 self assignee 붙이시고 머지해주시면 됩니다 |
확인했습니다..! |
:메모: 작업 내용
:링크: 관련 이슈
:말풍선: 추가 요청사항
:흰색_확인_표시: 체크리스트
코드 품질
테스트
배포 준비