fix: optimize log count calculation for large querysets

[rohan-chaturvedi]

🔍 Overview

The current resolver for secret logs computes a count using a standard COUNT operation. This becomes very expensive for large querysets with millions of rows. This calculation also blocks the return of log data, since it is computed within the same resolver.

💡 Proposed Changes

Use the SQL EXPLAIN command before computing a count on the total queryset. The number of rows affected based on the query plan can be used as an approximate count. If the expected count is below a certain threshold (10,000 rows by default), we can safely compute and return the actual count.

This means that counts above this threshold may not be exact, but do not significantly diverge given the simplicity of the query, and the accuracy of this count is not very relevant anyway. When the count is potentially not accurate, this is reflected by a ~ prefix

Additionally, the filter menu has been patched to correctly detected selected envs as an active filter

🖼️ Screenshots or Demo

💚 Did You...

• Ensure linting passes (code style checks)?
  - [ ] Update dependencies and lockfiles (if required)
  - [ ] Update migrations (if required)
  - [ ] Regenerate graphql schema and types (if required)
• Verify the app builds locally?
• Manually test the changes on different browsers/devices?

---

Note

Switch secret log counting to a fast approximate method with exact fallback and update the UI to show '~' for large counts and correctly detect environment filter usage.

• Backend:
  • Add api/utils/database.py#get_approximate_count leveraging PostgreSQL EXPLAIN to estimate row counts with exact fallback under a threshold (default 10000).
  • Replace secret_events_query.count() with get_approximate_count(secret_events_query) in backend/backend/schema.py (Query.resolve_secret_logs).
• Frontend:
  • In frontend/components/logs/SecretLogs.tsx, prefix event count with ~ when totalCount >= 10000 to signal approximation.
  • Fix hasActiveFilters to include selectedEnvironment as an active filter.

^{Written by Cursor Bugbot for commit 7ad5456. Configure here.}

[Copilot]

Pull Request Overview

This PR optimizes the count calculation for secret logs when dealing with large querysets by using PostgreSQL's EXPLAIN command to get approximate row counts instead of executing expensive COUNT operations on millions of rows. Counts below 10,000 rows still use exact counting for accuracy, while larger counts use estimates and are prefixed with "~" in the UI.

• Introduced get_approximate_count() utility function that uses EXPLAIN to estimate row counts
• Updated secret logs resolver to use approximate counting for better performance
• Fixed filter menu to properly detect selected environments as active filters

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
backend/api/utils/database.py	New utility function for approximate count calculation using EXPLAIN
backend/backend/schema.py	Replaced direct count() with get_approximate_count() in secret logs resolver
frontend/components/logs/SecretLogs.tsx	Added "~" prefix for approximate counts and fixed environment filter detection

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[nimish-ks]

@cursor review

[nimish-ks]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no bugs!

---