- Memory efficiency via
sync.Poolfor reusable buffers - Speed via worker pool and concurrent search
- Streaming results - batched indices sent as found
- Lazy loading - only fetch full entries when needed
- Early termination - metadata search first, skip full entry loads when possible
- Interface-based design - Builder pattern for extensibility
// motor/interfaces.go - NEW INTERFACES
// ReadRequest provides read-only access to entry read parameters
type ReadRequest interface {
GetOffset() int64
GetLength() int64
GetBuffer() *[]byte // May be nil (fallback to direct read)
}
// ReadRequestBuilder constructs ReadRequest instances with fluent API
type ReadRequestBuilder interface {
WithOffset(offset int64) ReadRequestBuilder
WithLength(length int64) ReadRequestBuilder
WithBuffer(buf *[]byte) ReadRequestBuilder
Build() ReadRequest
}
// ReadResponse provides read-only access to read results
type ReadResponse interface {
GetEntry() *harhar.Entry
GetBytesRead() int64
GetError() error
}
// EntryReader interface - REFACTORED to message-based
type EntryReader interface {
Read(ctx context.Context, req ReadRequest) ReadResponse
ReadMetadata(offset int64) (*EntryMetadata, error)
}// motor/reader_types.go - NEW FILE
// Private implementation of ReadRequest
type readRequest struct {
offset int64
length int64
buffer *[]byte
}
func (r *readRequest) GetOffset() int64 { return r.offset }
func (r *readRequest) GetLength() int64 { return r.length }
func (r *readRequest) GetBuffer() *[]byte { return r.buffer }
// Private builder implementation
type readRequestBuilder struct {
req readRequest
}
func NewReadRequestBuilder() ReadRequestBuilder {
return &readRequestBuilder{}
}
func (b *readRequestBuilder) WithOffset(offset int64) ReadRequestBuilder {
b.req.offset = offset
return b
}
func (b *readRequestBuilder) WithLength(length int64) ReadRequestBuilder {
b.req.length = length
return b
}
func (b *readRequestBuilder) WithBuffer(buf *[]byte) ReadRequestBuilder {
b.req.buffer = buf
return b
}
func (b *readRequestBuilder) Build() ReadRequest {
return &b.req
}
// Private implementation of ReadResponse
type readResponse struct {
entry *harhar.Entry
bytesRead int64
err error
}
func (r *readResponse) GetEntry() *harhar.Entry { return r.entry }
func (r *readResponse) GetBytesRead() int64 { return r.bytesRead }
func (r *readResponse) GetError() error { return r.err }
// Factory function for creating responses
func newReadResponse() *readResponse {
return &readResponse{}
}Benefits:
- Immutable requests once built
- Clean, fluent API for construction
- Future fields don't break existing code
- Private structs prevent direct access
- Extensible without signature changes
// motor/searcher.go
type SearchMode int
const (
PlainText SearchMode = iota
Regex
)
type SearchOptions struct {
Mode SearchMode // PlainText or Regex
SearchResponseBody bool // Deep search flag (default: false)
WorkerCount int // Default: runtime.NumCPU()
ChunkSize int // Entries per work batch (default: 0 = auto-partition)
}
// DefaultSearchOptions provides sensible defaults
var DefaultSearchOptions = SearchOptions{
Mode: PlainText,
SearchResponseBody: false,
WorkerCount: runtime.NumCPU(),
ChunkSize: 0, // Auto-partition
}Always Searches (Not Configurable):
- Metadata: URL, method, status, mimeType, serverIP
- Request headers (all)
- Response headers (all)
- Request body (POST data)
Design Rationale:
- Removed CaseSensitive (not needed initially)
- Removed SearchMetadata, SearchHeaders, SearchRequestBody (always enabled - no value in disabling)
- Only SearchResponseBody is optional (deep search can be expensive)
// motor/searcher.go
type SearchResult struct {
Index int // Entry index in HAR file
Field string // Which field matched: "url", "request.body", "response.headers.content-type"
Error error // Non-fatal error reading this entry (search continues)
}
type SearchStats struct {
EntriesSearched int64 // Total entries processed
MatchesFound int64 // Total matches found
BytesSearched int64 // Total bytes read from disk
SearchDuration time.Duration // Total search time
}
type atomicStats struct {
entriesSearched int64
matchesFound int64
bytesSearched int64
searchDuration int64 // Nanoseconds
} ┌──────────────┐
│ Consumer │
│ (TUI/CLI) │
└──────┬───────┘
│
│ Receives []SearchResult batches
↓
┌─────────────────────────────────────────┐
│ Result Channel (buffered) │
│ chan []SearchResult (buffer = Workers) │
└─────────────────────────────────────────┘
↑ ↑ ↑
│ │ │
┌──────┴────┬─────┴────┬─────┴──────┐
│ Worker 1 │ Worker 2 │ Worker N │
│ │ │ │
│ ┌──────┐ │ ┌──────┐ │ ┌──────┐ │
│ │Batch │ │ │Batch │ │ │Batch │ │ ← Accumulates matches
│ │Results│ │ │Results│ │ │Results│ │
│ └──────┘ │ └──────┘ │ └──────┘ │
│ │ │ │
│ ┌──────┐ │ ┌──────┐ │ ┌──────┐ │
│ │Buffer│ │ │Buffer│ │ │Buffer│ │ ← From sync.Pool
│ │64KB │ │ │64KB │ │ │64KB │ │
│ └──────┘ │ └──────┘ │ └──────┘ │
└──────┬────┴─────┬────┴─────┬──────┘
│ │ │
│ │ │ Pull work batches
↓ ↓ ↓
┌─────────────────────────────────────────┐
│ Work Queue (channel) │
│ chan workBatch (buffer = Workers * 2) │
└─────────────────────────────────────────┘
↑
│
┌────────┴─────────┐
│ Producer │
│ Creates batches │
│ [0-1000] │
│ [1000-2000] │
│ [2000-3000] │
│ ... │
└──────────────────┘
// motor/search_worker.go
type workBatch struct {
startIndex int // Inclusive start
endIndex int // Exclusive end (Go range convention)
}// motor/search_worker.go
func createWorkBatches(totalEntries int, opts SearchOptions) []workBatch {
var batches []workBatch
chunkSize := opts.ChunkSize
// Fallback: auto-partition based on worker count
if chunkSize == 0 {
chunkSize = (totalEntries + opts.WorkerCount - 1) / opts.WorkerCount
}
// Create batches
for start := 0; start < totalEntries; start += chunkSize {
end := start + chunkSize
if end > totalEntries {
end = totalEntries
}
batches = append(batches, workBatch{
startIndex: start,
endIndex: end,
})
}
return batches
}Example Work Distribution:
| Scenario | Batches Created |
|---|---|
| 50,000 entries, 10 workers, ChunkSize=0 | Auto: 5,000 entries/batch, 10 batches |
| 50,000 entries, 10 workers, ChunkSize=1000 | 1,000 entries/batch, 50 batches |
| 500 entries, 10 workers, ChunkSize=0 | 50 entries/batch, 10 batches |
Benefits:
- Configurable ChunkSize: User controls granularity
- Auto-partition fallback: Smart default based on worker count
- Dynamic load balancing: Workers pull batches from queue as available
// motor/search_worker.go
func worker(ctx context.Context,
workQueue <-chan workBatch,
results chan<- []SearchResult,
searcher *HARSearcher,
pattern compiledPattern,
opts SearchOptions) {
for {
select {
case <-ctx.Done():
return
case batch, ok := <-workQueue:
if !ok {
return // Work queue closed, all done
}
// Get buffer from pool ONCE per batch
buf := searcher.bufferPool.Get().(*[]byte)
// Accumulate matches (zero initial capacity, grows as needed)
var batchResults []SearchResult
// Process each entry in this batch
// Context only checked when pulling work (not inside loop)
for i := batch.startIndex; i < batch.endIndex; i++ {
result := searchEntry(ctx, searcher, i, pattern, opts, buf)
if result != nil {
batchResults = append(batchResults, *result)
}
atomic.AddInt64(&searcher.stats.entriesSearched, 1)
}
// Return buffer to pool immediately
searcher.bufferPool.Put(buf)
// Send batch results if any matches found
if len(batchResults) > 0 {
select {
case results <- batchResults:
atomic.AddInt64(&searcher.stats.matchesFound, int64(len(batchResults)))
case <-ctx.Done():
return
}
}
}
}
}Key Design Decisions:
- ✅ No context check inside batch loop: Only checked when pulling work from queue
- ✅ Zero initial capacity for results: Let Go's append grow as needed
- ✅ Single buffer per batch: Retrieved once, used for all entries, returned immediately
- ✅ Batch result delivery: Accumulate matches, send once per batch
- ✅ Workers complete current batch: Even if context cancelled mid-batch
// motor/search_worker.go
func searchEntry(ctx context.Context,
s *HARSearcher,
index int,
pattern compiledPattern,
opts SearchOptions,
buf *[]byte) *SearchResult {
// STEP 1: Metadata search (NO I/O - instant lookup)
metadata, err := s.streamer.GetMetadata(index)
if err != nil {
return &SearchResult{Index: index, Error: err}
}
// Search metadata fields (always enabled)
if matches(metadata.URL, pattern) {
return &SearchResult{Index: index, Field: "url"} // EARLY RETURN - skip loading entry!
}
if matches(metadata.Method, pattern) {
return &SearchResult{Index: index, Field: "method"} // EARLY RETURN
}
if matches(metadata.StatusText, pattern) {
return &SearchResult{Index: index, Field: "status"} // EARLY RETURN
}
if matches(metadata.MimeType, pattern) {
return &SearchResult{Index: index, Field: "mimeType"} // EARLY RETURN
}
if matches(metadata.ServerIP, pattern) {
return &SearchResult{Index: index, Field: "serverIP"} // EARLY RETURN
}
// STEP 2: Only load full entry if metadata didn't match (lazy loading)
req := NewReadRequestBuilder().
WithOffset(metadata.FileOffset).
WithLength(metadata.Length).
WithBuffer(buf). // Pooled buffer for efficiency
Build()
// STEP 3: Read entry using pooled buffer
resp := s.reader.Read(ctx, req)
if resp.GetError() != nil {
return &SearchResult{Index: index, Error: resp.GetError()}
}
entry := resp.GetEntry()
atomic.AddInt64(&s.stats.bytesSearched, resp.GetBytesRead())
// STEP 4: Search request headers (always enabled)
for _, header := range entry.Request.Headers {
if matches(header.Name, pattern) || matches(header.Value, pattern) {
return &SearchResult{Index: index, Field: "request.headers." + header.Name}
}
}
// STEP 5: Search request body (always enabled)
if entry.Request.PostData != nil && entry.Request.PostData.Text != "" {
if matches(entry.Request.PostData.Text, pattern) {
return &SearchResult{Index: index, Field: "request.body"}
}
}
// STEP 6: Search response headers (always enabled)
for _, header := range entry.Response.Headers {
if matches(header.Name, pattern) || matches(header.Value, pattern) {
return &SearchResult{Index: index, Field: "response.headers." + header.Name}
}
}
// STEP 7: Search response body (OPTIONAL - deep search)
if opts.SearchResponseBody && entry.Response.Content.Text != "" {
if matches(entry.Response.Content.Text, pattern) {
return &SearchResult{Index: index, Field: "response.body"}
}
}
return nil // No match
}Search Strategy:
- Metadata first: Fast O(1) lookup, no I/O
- Early return: First match exits immediately
- Lazy loading: Only load full entry if metadata doesn't match
- Deep search optional: Response bodies only if flag enabled
- Pooled buffer usage: All file I/O uses pre-allocated buffer
// motor/reader.go - MODIFIED
type DefaultEntryReader struct {
filePath string
filePool *sync.Pool // NEW: Pool of *os.File handles for concurrent access
index *Index
offsetIndex map[int64]*EntryMetadata // NEW: O(1) metadata lookup by offset
}
func NewEntryReader(filePath string, index *Index) (*DefaultEntryReader, error) {
// Build offset index once during initialization (O(n) → O(1) lookups)
offsetIndex := make(map[int64]*EntryMetadata, len(index.Entries))
for i := range index.Entries {
offsetIndex[index.Entries[i].FileOffset] = index.Entries[i]
}
// Create file handle pool
filePool := &sync.Pool{
New: func() interface{} {
file, err := os.Open(filePath)
if err != nil {
return nil // Handle error in Read()
}
return file
},
}
return &DefaultEntryReader{
filePath: filePath,
filePool: filePool,
index: index,
offsetIndex: offsetIndex,
}, nil
}
func (r *DefaultEntryReader) Read(ctx context.Context, req ReadRequest) ReadResponse {
resp := newReadResponse()
// Get file handle from pool (no mutex needed - each worker gets own handle)
fileHandle := r.filePool.Get().(*os.File)
if fileHandle == nil {
resp.err = fmt.Errorf("failed to open file handle")
return resp
}
defer r.filePool.Put(fileHandle) // Return to pool
// Seek to offset
_, err := fileHandle.Seek(req.GetOffset(), io.SeekStart)
if err != nil {
resp.err = fmt.Errorf("seek failed: %w", err)
return resp
}
// Create limited reader for exact byte count
limitReader := io.LimitReader(fileHandle, req.GetLength())
// Use pooled buffer if provided, otherwise fallback to direct read
var jsonReader io.Reader
if buf := req.GetBuffer(); buf != nil {
// SEARCH PATH: Use pooled buffer for maximum efficiency
// Validate buffer size
if req.GetLength() > int64(cap(*buf)) {
// Buffer too small, grow it
*buf = make([]byte, req.GetLength())
}
n, err := io.ReadFull(limitReader, (*buf)[:req.GetLength()])
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
resp.err = fmt.Errorf("read failed: %w", err)
return resp
}
resp.bytesRead = int64(n)
// Create reader from buffer slice
jsonReader = bytes.NewReader((*buf)[:n])
} else {
// FALLBACK PATH: TUI/serve use cases (not search)
// Direct read without buffer - less efficient but backward compatible
jsonReader = limitReader
resp.bytesRead = req.GetLength() // Approximate
}
// Skip leading JSON array separators (commas, whitespace)
skipReader := &skipLeadingReader{r: jsonReader}
// Decode entry from JSON
decoder := json.NewDecoder(skipReader)
var entry harhar.Entry
if err := decoder.Decode(&entry); err != nil {
resp.err = fmt.Errorf("decode failed: %w", err)
return resp
}
resp.entry = &entry
return resp
}
// ReadMetadata with O(1) lookup using offset index map
func (r *DefaultEntryReader) ReadMetadata(offset int64) (*EntryMetadata, error) {
if meta, ok := r.offsetIndex[offset]; ok {
return meta, nil
}
return nil, fmt.Errorf("metadata not found for offset %d", offset)
}Key Points:
- ✅ File handle pool: Eliminates mutex contention, enables true parallel I/O
- ✅ O(1) metadata lookup: Offset index map replaces O(n) linear scan
- ✅ Buffer size validation: Dynamically grows buffer if entry exceeds 64KB
- ✅ Builder pattern for requests: Clean, extensible API
- ✅ Buffer fallback preserved: Backward compatible with TUI/serve
- ✅ Exact byte tracking: Response reports actual bytes read
// motor/search_pattern.go - NEW FILE
type compiledPattern struct {
mode SearchMode
plainText string
regex *regexp.Regexp
caseSensitive bool // Reserved for future use
}
func compilePattern(pattern string, opts SearchOptions) (compiledPattern, error) {
cp := compiledPattern{
mode: opts.Mode,
caseSensitive: true, // Hardcoded for now (removed from opts)
}
if opts.Mode == Regex {
// Compile regex pattern
regex, err := regexp.Compile(pattern)
if err != nil {
return cp, fmt.Errorf("invalid regex pattern: %w", err)
}
cp.regex = regex
} else {
// Plain text pattern
cp.plainText = pattern
}
return cp, nil
}
func matches(haystack string, pattern compiledPattern) bool {
if pattern.mode == Regex {
return pattern.regex.MatchString(haystack)
}
// Plain text: use strings.Contains (faster than regex)
return strings.Contains(haystack, pattern.plainText)
}Performance:
- Pattern compiled once before search starts
- Plain text uses
strings.Contains(optimized assembly on many platforms) - Regex uses compiled
regexp.Regexp(no re-compilation per entry)
// motor/searcher.go
type HARSearcher struct {
streamer HARStreamer
reader EntryReader
bufferPool *sync.Pool
stats atomicStats
}
func NewSearcher(streamer HARStreamer) *HARSearcher {
// Note: May need to expose reader via interface method instead of type assertion
defaultStreamer, ok := streamer.(*DefaultHARStreamer)
if !ok {
panic("NewSearcher requires DefaultHARStreamer")
}
return &HARSearcher{
streamer: streamer,
reader: defaultStreamer.reader,
bufferPool: &sync.Pool{
New: func() interface{} {
buf := make([]byte, 64*1024) // 64KB buffers
return &buf
},
},
}
}
func (s *HARSearcher) Search(ctx context.Context, pattern string, opts SearchOptions) (<-chan []SearchResult, error) {
// Set defaults
if opts.WorkerCount == 0 {
opts.WorkerCount = runtime.NumCPU()
}
// Compile pattern ONCE (not per entry!)
compiledPattern, err := compilePattern(pattern, opts)
if err != nil {
return nil, fmt.Errorf("invalid pattern: %w", err)
}
// Get total entries
index := s.streamer.GetIndex()
totalEntries := index.TotalEntries
// Handle empty HAR
if totalEntries == 0 {
emptyResults := make(chan []SearchResult)
close(emptyResults)
return emptyResults, nil
}
// Create work batches
batches := createWorkBatches(totalEntries, opts)
// Create channels
workQueue := make(chan workBatch, opts.WorkerCount*2)
results := make(chan []SearchResult, opts.WorkerCount)
// Start timer
startTime := time.Now()
// Spawn fixed worker pool
var wg sync.WaitGroup
for i := 0; i < opts.WorkerCount; i++ {
wg.Add(1)
go func() {
defer wg.Done()
worker(ctx, workQueue, results, s, compiledPattern, opts)
}()
}
// Producer goroutine
go func() {
defer close(workQueue)
for _, batch := range batches {
select {
case workQueue <- batch:
case <-ctx.Done():
return
}
}
}()
// Collector goroutine
go func() {
wg.Wait() // Wait for all workers to finish
close(results) // Signal consumer: no more results
// Record final stats
duration := time.Since(startTime)
atomic.StoreInt64(&s.stats.searchDuration, int64(duration))
}()
return results, nil
}
func (s *HARSearcher) Stats() SearchStats {
return SearchStats{
EntriesSearched: atomic.LoadInt64(&s.stats.entriesSearched),
MatchesFound: atomic.LoadInt64(&s.stats.matchesFound),
BytesSearched: atomic.LoadInt64(&s.stats.bytesSearched),
SearchDuration: time.Duration(atomic.LoadInt64(&s.stats.searchDuration)),
}
}For 50,000 entries, 10 workers, ChunkSize=1000:
| Component | Memory Usage |
|---|---|
| Index (pre-loaded) | ~9.6 MB |
| Worker goroutines (10 × 2KB stack) | 20 KB |
| Buffer pool (10 × 64KB) | 640 KB |
| Work queue (20 batches buffered) | ~320 bytes |
| Result queue (10 result batches) | ~2 KB |
| Entries in flight (10 × ~200KB avg) | ~2 MB |
| Total | ~12.3 MB |
Heap Allocation Savings:
- Without buffer pool: 50,000 entries × 200KB = 10 GB heap pressure
- With buffer pool: 10 workers × 64KB = 640 KB (reused)
- Savings: ~99.99% reduction in heap allocations
Assumptions:
- 50,000 entries
- 10 workers
- ~240μs per entry read (from motor benchmarks)
- ~50μs per pattern match
Standard Search (metadata + headers + request body):
- Sequential: 50,000 × 290μs = 14.5 seconds
- Parallel (10 workers): 14.5s ÷ 10 = ~1.5 seconds
Deep Search (including response bodies, avg 100KB per body):
- Sequential: 50,000 × 2.3ms = 115 seconds
- Parallel (10 workers): 115s ÷ 10 = ~11.5 seconds
Scalability:
- Scales linearly with worker count (up to I/O saturation)
- Bounded memory regardless of entry count
- Context cancellation allows early termination
Files to modify:
motor/types.go- Add new types if neededmotor/interfaces.go- Add ReadRequest/ReadResponse/ReadRequestBuilder interfacesmotor/reader_types.go- NEW: Implement builder patternmotor/reader.go- Update DefaultEntryReader.Read() signature
Migration required:
- Update all existing callers in TUI
- Update serve command
- Update any tests
New files:
motor/searcher.go- HARSearcher, SearchOptions, SearchResult typesmotor/search_worker.go- Worker pool, searchEntry, createWorkBatchesmotor/search_pattern.go- Pattern compilation and matchingmotor/searcher_test.go- Unit tests
Tests to write:
- Pattern compilation (plain text and regex)
- Work batch creation (various sizes and chunk configurations)
- Worker pool execution
- Result batching
- Buffer pool usage
- Context cancellation
- Error handling
CLI command:
cmd/search.go- New search subcommand- Flags:
--regex,--deep,--workers,--chunk-size - Output formatting: table or JSON
TUI integration:
- Add search mode keybinding (e.g.,
/for search) - Search input field
- Search results view (filtered table)
- Progress indicator
- Cancel search on ESC
Performance tests:
- Benchmark small HAR (1,000 entries)
- Benchmark medium HAR (10,000 entries)
- Benchmark large HAR (100,000+ entries)
- Memory profiling
- Compare plain text vs regex performance
// Initialize streamer
streamer, err := motor.NewHARStreamer("capture.har", motor.DefaultStreamerOptions())
if err != nil {
log.Fatal(err)
}
if err := streamer.Initialize(context.Background()); err != nil {
log.Fatal(err)
}
defer streamer.Close()
// Create searcher
searcher := motor.NewSearcher(streamer)
// Execute search with defaults
opts := motor.DefaultSearchOptions
resultChan, err := searcher.Search(context.Background(), "api-key", opts)
if err != nil {
log.Fatal(err)
}
// Consume results
totalMatches := 0
for resultBatch := range resultChan {
for _, result := range resultBatch {
if result.Error != nil {
log.Printf("Error at entry %d: %v", result.Index, result.Error)
continue
}
fmt.Printf("Match at entry %d in field: %s\n", result.Index, result.Field)
totalMatches++
}
}
fmt.Printf("Found %d total matches\n", totalMatches)ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Configure search
opts := motor.SearchOptions{
Mode: motor.Regex,
SearchResponseBody: true, // Deep search
WorkerCount: 8,
ChunkSize: 500, // 500 entries per batch
}
// Start search
resultChan, err := searcher.Search(ctx, `api[._-]?key`, opts)
if err != nil {
log.Fatal(err)
}
// Progress ticker
ticker := time.NewTicker(500 * time.Millisecond)
defer ticker.Stop()
// Consume results with progress updates
totalMatches := 0
done := false
for !done {
select {
case resultBatch, ok := <-resultChan:
if !ok {
done = true
break
}
for _, result := range resultBatch {
if result.Error == nil {
fmt.Printf("Match: entry %d, field %s\n", result.Index, result.Field)
totalMatches++
}
}
case <-ticker.C:
stats := searcher.Stats()
index := searcher.streamer.GetIndex()
progress := float64(stats.EntriesSearched) / float64(index.TotalEntries) * 100
fmt.Printf("Progress: %.1f%% (%d/%d entries)\n",
progress, stats.EntriesSearched, index.TotalEntries)
}
}
// Final stats
stats := searcher.Stats()
fmt.Printf("\nSearch complete:\n")
fmt.Printf(" Entries searched: %d\n", stats.EntriesSearched)
fmt.Printf(" Matches found: %d\n", stats.MatchesFound)
fmt.Printf(" Bytes read: %d MB\n", stats.BytesSearched/1024/1024)
fmt.Printf(" Duration: %v\n", stats.SearchDuration)ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
resultChan, err := searcher.Search(ctx, "pattern", motor.DefaultSearchOptions)
// User can press Ctrl+C to cancel
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt)
go func() {
<-sigChan
fmt.Println("\nCancelling search...")
cancel()
}()
// Consume results until cancelled or complete
for resultBatch := range resultChan {
// Process results...
}
stats := searcher.Stats()
fmt.Printf("Search cancelled after %v\n", stats.SearchDuration)✅ Interface-based design
- ReadRequest/ReadResponse as interfaces
- Builder pattern for immutable, fluent construction
- Private structs, public interfaces
- Future-proof extensibility without breaking changes
✅ Simplified configuration
- Removed unnecessary options (CaseSensitive, SearchMetadata, etc.)
- Only essential options remain
- Always searches metadata, headers, request bodies
- Only response body search is optional
✅ Efficient worker pool
- Fixed goroutines (bounded concurrency)
- Context checked only at work queue boundaries
- Zero initial batch capacity, grows dynamically as needed
- Single pooled buffer per worker
- File handle pool eliminates mutex contention
✅ Smart work distribution
- Configurable ChunkSize with auto-partition fallback
- Dynamic load balancing via work queue
- Scales from small to massive HAR files
✅ Batch result delivery
- Workers accumulate matches per batch
- Reduces channel operations
- Consumer receives results in chunks
- Lower overhead than per-match streaming
✅ Memory efficiency
- sync.Pool for 64KB buffers
- ~99.99% reduction in heap allocations
- Bounded memory usage regardless of HAR size
- Buffer reuse across all entries in batch
✅ Backward compatibility
- Buffer fallback in Read() for TUI/serve
- Search always provides buffer
- Existing code continues to work
- Migration path is clear
motor/
reader_types.go - ReadRequest/ReadResponse implementations with builder
searcher.go - HARSearcher, SearchOptions, SearchResult, SearchStats
search_worker.go - Worker pool, searchEntry, createWorkBatches
search_pattern.go - Pattern compilation and matching (PlainText/Regex)
searcher_test.go - Unit tests and benchmarks
motor/
interfaces.go - Add ReadRequest/ReadResponse/ReadRequestBuilder interfaces
reader.go - Update DefaultEntryReader.Read() to use new interface
types.go - Add any new constants
cmd/
search.go - CLI search command with flags
tui/
search_view.go - Search mode UI
search_results.go - Search results table view
- Cache frequently accessed entries (LRU cache)
- Parallel file I/O with file handle pool (IMPLEMENTED)
- Memory-mapped file access for very large HARs
- Offset-indexed map for O(1) metadata lookup (IMPLEMENTED)
- Full-text search index for repeated searches
- Search result ranking/scoring
- Highlight matched text in TUI
- Export search results to new HAR file
- Save/load search patterns
- Search history
- Multi-pattern search (AND/OR/NOT logic)
- Expose EntryReader via HARStreamer interface method
- Add SearchResult.MatchPosition for snippet extraction
- Progress callback instead of polling Stats()
- Streaming stats updates over channel
This plan includes the following critical performance and correctness fixes:
- Problem: Single file handle with mutex lock serialized all I/O operations
- Solution: File handle pool (
sync.Pool) gives each worker its own handle - Impact: Enables true parallel reads, eliminates mutex contention bottleneck
- Problem:
ReadMetadata(offset)performed O(n) linear scan through all entries - Solution: Build
map[int64]*EntryMetadataoffset index during initialization - Impact: Changes 50,000 lookups from 62 seconds to <0.001 seconds
- Problem: Fixed 64KB buffers would panic on entries > 64KB
- Solution: Validate buffer capacity and dynamically grow if needed
- Impact: Handles entries of any size safely
- Problem: Metadata search always followed by full entry load (defeating purpose)
- Solution: Added explicit early returns when metadata matches
- Impact: Skips expensive I/O for ~20-30% of searches (URL/status matches)
- Problem: Arbitrary constants (min 100 batch size, capacity 8 for results)
- Solution: Removed minimum batch size, use zero initial capacity for results
- Impact: Cleaner code, let Go's append handle growth naturally
- Problem: Unused
GetFields()/GetPartial()methods added complexity - Solution: Removed from ReadRequest and ReadResponse interfaces
- Impact: Simpler API, less confusion
This architecture provides:
- Speed: Worker pool with true parallel I/O (file handle pool)
- Efficiency: Buffer pooling, lazy loading, real early termination
- Performance: O(1) metadata lookups, no mutex contention
- Flexibility: Configurable workers, chunk size, search depth
- Maintainability: Clean interfaces, builder pattern, separation of concerns
- Extensibility: Easy to add features without breaking changes
Next step: Begin Phase 1 implementation with EntryReader refactor.