|
| 1 | +# SmartCSV: A Parallel CSV Processing Library for Spring Boot |
| 2 | +[](https://jitpack.io/#saadzarook/smartcsv) |
| 3 | +[](LICENSE) |
| 4 | +[](https://github.com/saadzarook/smartcsv/stargazers) |
| 5 | +[](https://github.com/saadzarook/smartcsv/network) |
| 6 | +[](https://github.com/saadzarook/smartcsv/issues) |
| 7 | + |
| 8 | +## Introduction |
| 9 | + |
| 10 | +**SmartCSV** is a Spring Boot library designed to simplify the handling of CSV file uploads and processing. It allows developers to: |
| 11 | + |
| 12 | +- **Upload CSV Files Easily**: Streamline CSV file uploads via REST endpoints. |
| 13 | +- **Validate and Process Data in Parallel**: Improve performance by concurrently validating and processing records. |
| 14 | +- **Define Custom Validations**: Specify custom validation rules for CSV headers and data fields. |
| 15 | +- **Flexible Error Handling**: Choose how to handle validation errors—skip records, stop processing, or collect and return all errors. |
| 16 | +- **Annotation-Based Configuration**: Integrate seamlessly into existing projects with minimal effort. |
| 17 | + |
| 18 | +## Features |
| 19 | + |
| 20 | +- **Annotation-Based Setup**: Use `@SmartCsvProcessor` and `@SmartCsvField` annotations to configure processing. |
| 21 | +- **Custom Validations**: Implement your own validation logic for headers and records. |
| 22 | +- **Parallel Processing**: Utilize multi-threading to process large datasets efficiently. |
| 23 | +- **Flexible Error Handling**: Decide whether to skip invalid records, halt processing on errors, or collect errors for reporting. |
| 24 | +- **Detailed Logging**: Monitor processing steps and errors through comprehensive SLF4J logging. |
| 25 | + |
| 26 | +## Installation |
| 27 | + |
| 28 | +Add the following dependency to your project's `pom.xml` or `build.gradle` file: |
| 29 | +This project uses JitPack to host the library. To add the dependency to your project, follow the instructions below: |
| 30 | +**For Maven (`pom.xml`):** |
| 31 | + |
| 32 | +```xml |
| 33 | +<repositories> |
| 34 | + <repository> |
| 35 | + <id>jitpack.io</id> |
| 36 | + <url>https://jitpack.io</url> |
| 37 | + </repository> |
| 38 | +</repositories> |
| 39 | +``` |
| 40 | +```xml |
| 41 | +<dependency> |
| 42 | +<groupId>com.github.saadzarook</groupId> |
| 43 | +<artifactId>smartcsv</artifactId> |
| 44 | +<version>v1.0.0</version> |
| 45 | +</dependency> |
| 46 | + |
| 47 | +``` |
| 48 | + |
| 49 | +For Gradle (`build.gradle`): |
| 50 | + |
| 51 | +``` |
| 52 | +repositories { |
| 53 | + mavenCentral() |
| 54 | + maven { url 'https://jitpack.io' } |
| 55 | +} |
| 56 | +
|
| 57 | +dependencies { |
| 58 | + implementation 'com.github.saadzarook:smartcsv:v1.0.0' |
| 59 | +} |
| 60 | +``` |
| 61 | + |
| 62 | +## Usage |
| 63 | + |
| 64 | +1. Annotate your processing class with `@SmartCsvProcessor` |
| 65 | +```java |
| 66 | +@SmartCsvProcessor( |
| 67 | + model = User.class, |
| 68 | + validationStrategy = "collect", // Options: skip, stop, collect |
| 69 | + headerValidator = CustomHeaderValidator.class // Optional |
| 70 | +) |
| 71 | +public void processUser(User user) { |
| 72 | + // Your processing logic here |
| 73 | + userService.save(user); |
| 74 | +} |
| 75 | +``` |
| 76 | +2. Define Your Model Class |
| 77 | +```java |
| 78 | +public class User { |
| 79 | + @SmartCsvField(header = "Name", required = true) |
| 80 | + private String name; |
| 81 | + |
| 82 | + @SmartCsvField(header = "Email", required = true, validation = "[^@ ]+@[^@ ]+\\.[^@ ]+") |
| 83 | + private String email; |
| 84 | + |
| 85 | + // Getters and setters |
| 86 | +} |
| 87 | +``` |
| 88 | +3. Implement a Custom Header Validator (Optional) |
| 89 | +```java |
| 90 | +public class CustomHeaderValidator implements HeaderValidator { |
| 91 | + @Override |
| 92 | + public boolean validateHeaders(String[] headers) { |
| 93 | + return Arrays.asList(headers).contains("Name") && Arrays.asList(headers).contains("Email"); |
| 94 | + } |
| 95 | +} |
| 96 | +``` |
| 97 | +4. Handle CSV File Uploads via REST Endpoint |
| 98 | +```java |
| 99 | +@RestController |
| 100 | +@RequestMapping("/users") |
| 101 | +public class UserController { |
| 102 | + |
| 103 | + @Autowired |
| 104 | + private CsvProcessingService csvProcessingService; |
| 105 | + |
| 106 | + @PostMapping("/upload") |
| 107 | + public ResponseEntity<?> uploadCsv(@RequestParam("file") MultipartFile file) { |
| 108 | + try { |
| 109 | + // Get the method annotated with @SmartCsvProcessor |
| 110 | + Method processUserMethod = this.getClass().getMethod("processUser", User.class); |
| 111 | + |
| 112 | + // Process the CSV file |
| 113 | + List<RecordError> errors = csvProcessingService.processCsv(file, processUserMethod, this); |
| 114 | + |
| 115 | + if (!errors.isEmpty()) { |
| 116 | + // Return errors to the client |
| 117 | + return ResponseEntity.badRequest().body(errors); |
| 118 | + } |
| 119 | + |
| 120 | + return ResponseEntity.ok("CSV file processed successfully"); |
| 121 | + } catch (Exception e) { |
| 122 | + // Handle exceptions |
| 123 | + return ResponseEntity.status(500).body("Internal Server Error"); |
| 124 | + } |
| 125 | + } |
| 126 | + |
| 127 | + @SmartCsvProcessor( |
| 128 | + model = User.class, |
| 129 | + validationStrategy = "collect", |
| 130 | + headerValidator = CustomHeaderValidator.class |
| 131 | + ) |
| 132 | + public void processUser(User user) { |
| 133 | + // Processing logic |
| 134 | + userService.save(user); |
| 135 | + } |
| 136 | +} |
| 137 | +``` |
| 138 | + |
| 139 | +## Error Handling Strategies |
| 140 | +Specify how the library should handle validation errors using the validationStrategy parameter: |
| 141 | + |
| 142 | +- skip: Skip invalid records and continue processing. |
| 143 | +- stop: Stop processing when an error is encountered. |
| 144 | +- collect: Collect all errors and return them after processing. |
| 145 | + |
| 146 | +## Contribution |
| 147 | +Contributions are welcome! Feel free to open issues or submit pull requests to help improve this library. |
| 148 | + |
| 149 | +## License |
| 150 | +This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. |
| 151 | + |
| 152 | +## Contact |
| 153 | +For questions or support, please contact Saad Zarook at [email protected] |
0 commit comments