Skip to content

Commit b11b5e2

Browse files
committed
Replace IOPS calculations with benchmarking tool reference
- Remove IOPS calculation section with hardcoded throughput numbers - Remove scaling guidelines with specific node/throughput estimates - Add reference to forthcoming BigQuery Ingestor Benchmarking Tool - Update troubleshooting to recommend benchmarking instead of IOPS checks - Encourage users to use proper benchmarking for their specific workloads
1 parent 5a1f8bf commit b11b5e2

File tree

1 file changed

+10
-14
lines changed

1 file changed

+10
-14
lines changed

README.md

Lines changed: 10 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -509,7 +509,7 @@ LIMIT 10;
509509

510510
1. **Increase batch sizes** in config for faster catch-up
511511
2. **Add more nodes** to the cluster for horizontal scaling
512-
3. **Check IOPS capacity** - ensure storage can handle write throughput
512+
3. **Benchmark your workload** - use the benchmarking tool to determine optimal cluster sizing
513513
4. **Reduce columns** - fetch only needed columns to reduce network transfer
514514

515515
### Data Drift Detected
@@ -554,24 +554,20 @@ LIMIT 10;
554554

555555
## Performance Tuning
556556

557-
### IOPS Calculation
557+
### Benchmarking Your Workload
558558

559-
```
560-
Indexes: 1 primary + 1 timestamp = 2 indexes
561-
IOPS per record: ~4 IOPS
562-
Target throughput: 5000 records/sec per node
563-
Required IOPS: 20,000 per node
564-
```
559+
To determine the optimal cluster size and configuration for your specific use case, use the **BigQuery Ingestor Benchmarking Tool** (coming soon). The tool will:
565560

566-
Learn more about [Harper's storage architecture](https://docs.harperdb.io/docs/reference/storage-algorithm)
561+
- Measure actual throughput with your data volume and record sizes
562+
- Test different batch size configurations
563+
- Recommend optimal cluster sizing based on your target latency
564+
- Identify storage and network bottlenecks specific to your workload
567565

568-
### Scaling Guidelines
566+
Until the benchmarking tool is available, start with the batch size recommendations below and monitor your sync lag to determine if scaling is needed.
569567

570-
- **3 nodes**: ~15K records/sec total
571-
- **6 nodes**: ~30K records/sec total
572-
- **12 nodes**: ~60K records/sec total
568+
**Note:** Harper doesn't autoscale. Add/remove nodes manually via Fabric UI or self-hosted configuration. Cluster size changes require workload rebalancing (see Limitations).
573569

574-
**Note:** Harper doesn't autoscale. Add/remove nodes manually via Fabric UI or self-hosted configuration. Cluster size changes require consideration (see Limitations).
570+
Learn more about [Harper's storage architecture](https://docs.harperdb.io/docs/reference/storage-algorithm)
575571

576572
### Batch Size Recommendations
577573

0 commit comments

Comments
 (0)