[SR. NO. 21] Fix Face Recognition Performance Bottleneck with Smart Clustering and Database Enhancements

[priyankeshh]

Overview

This PR addresses the critical performance bottleneck in face recognition clustering while introducing comprehensive database optimizations and monitoring capabilities.

Core Changes

Performance Optimizations

• Implemented smart batch processing for face embeddings
• Added incremental clustering strategy
• Optimized memory usage through efficient data structures
• Introduced database indexing for faster queries

Database Enhancements

# Key database improvements:
1. Context-managed connections
2. Batch operations support
3. Automated cleanup routines
4. Query optimization with GROUP_CONCAT
5. Performance monitoring utilities

Memory Management

• Implemented configurable batch sizes
• Added memory-aware processing
• Optimized database operations for large datasets

Technical Details

Database Operations

def batch_insert_face_embeddings(
    embeddings_batch: List[Tuple[str, List[np.ndarray]]],
    batch_size: int = 1000
) -> None:
    """
    Efficient batch processing with memory optimization
    """

Performance Monitoring

def get_database_stats() -> dict:
    """
    Real-time performance metrics and database health monitoring
    """

📊 Performance Test Results

Key Performance Metrics

Test Configuration

• 100 images
• 5 embeddings per image
• 128-dimensional embeddings
• 1,000 batch size

Breaking Changes

None. All changes are backward compatible.

Checklist

• Performance bottleneck resolved (98.1% improvement)
• Database optimizations implemented
• Memory management improved (minimal 3.5% increase)
• Test results documented
• Code reviewed

Fixes #350

[priyankeshh]

@Pranav0-0Aggarwal Would you please be able to review this PR when you have a moment? I've been keeping it up to date with the main branch, and it's ready for final review. Thank you !

[Jibesh10101011]

Hey @priyankeshh, why does this method batch_insert_face_embeddings ?

def batch_insert_face_embeddings(
    embeddings_batch: List[Tuple[str, List[np.ndarray]]],
    batch_size: int = 1000
) -> None:
    """
    Efficient batch processing with memory optimization
    """

As In the actual application, the face-tagging algorithm processes only a single image_path when a new image is added:
GitHub Link

So, at the moment, there doesn’t seem to be a use case for batch_insert_face_embeddings.

On what basis, then, were these 📊 Performance Test Results & Key Performance Metrics derived?

[priyankeshh]

Hey @Jibesh10101011

So about the batch insertion - you're right that the current flow processes images one by one. The thing is, I noticed while testing that even though we process images individually, the bottleneck happens when these operations stack up.

What was happening is that after every few images (around 5-6), the system would trigger a full reclustering operation with DBSCAN. This gets exponentially slower as more images are added to the system. So even though we're adding images one at a time, the clustering was becoming a major bottleneck.

The batch insertion function is partly forward-looking - I'm planning to update the scheduler in a follow-up PR to better handle bulk operations. But even with the current flow, it helps optimize how we handle the database connections and reduces the clustering overhead.

For the performance metrics, I wrote a test script that simulates adding multiple images to the system. It basically replicates what happens when users import a folder or when multiple images get processed in sequence. The 98% improvement comes mainly from avoiding unnecessary reclustering operations and optimizing the database interactions.
You can see the test_performance.py

[Jibesh10101011]

The batch insertion function is partly forward-looking - I'm planning to update the scheduler in a follow-up PR to better handle bulk operations. But even with the current flow, it helps optimize how we handle the database connections and reduces the clustering overhead.

I see your point about optimizing database connections, but this batch insertion approach is only beneficial when processing multiple images at once. However, in our current architecture, each image is first checked for its type—if it's a 'person,' only then does it proceed to clustering. This check happens specifically when handling @router.route("/images/add-folder").

Now, if I modify the architecture as per your suggestion, let’s analyze the impact. Suppose we have 1000 images, and 500 of them are of type 'person.' Your approach suggests inserting these 500 images in a batch. However, after insertion, all these images still need to be compared with existing face embeddings for clustering. This means the linear expansion issue persists—batching doesn't eliminate the fundamental scaling problem.

[priyankeshh]

Oh wait, I see what you're saying now. You're absolutely right - I got so focused on optimizing the database operations that I missed the bigger bottleneck in the clustering algorithm itself.

I fixed the problem locally and I will updating this PR soon

[priyankeshh]

I've now implemented incremental clustering to address the performance bottleneck:

Before:

1. Every new face embedding triggered a full DBSCAN reclustering
2. O(n²) complexity made processing slow with larger datasets
3. No batch processing support for multiple faces

Now:

1. New approach avoids rerunning DBSCAN on the entire dataset for each new image
2. Uses vectorized distance calculations to efficiently compare new embeddings only to existing clusters
3. Only assigns new cluster IDs when necessary
4. Added proper batch processing support in add_faces_batch()

Added test file:

1. Created tests/test_incremental_clustering.py to benchmark performance

• Tests Result:
  

The changes maintain clustering accuracy while dramatically improving performance. This addresses the O(n²) complexity problem you pointed out, making the application responsive even with large image collections.

Thank you @Jibesh10101011 for pointing this out

[priyankeshh]

@Pranav0-0Aggarwal can you take a look at this PR?