Skip to content

Conversation

albertvillanova
Copy link
Member

@albertvillanova albertvillanova commented Oct 17, 2025

Optimize comprehension evaluation with generator-based approach in LocalPythonExecutor.

This PR refactors the comprehension evaluation logic to use a generator-based approach for better performance and memory efficiency.

Performance Benefits

  1. Memory efficiency: Eliminates intermediate list allocations during comprehension evaluation
  2. Lazy evaluation: Only computes what's needed, especially beneficial for set() and dict() where duplicates are discarded
  3. Optimized construction: list(generator) and set(generator) are highly optimized in CPython
  4. Reduced overhead: No more list wrapping/unwrapping in base case evaluators

Memory Complexity Improvement

The refactored implementation reduces memory overhead from O(n) to O(1) for the iteration mechanism itself:

  • Before: Built complete intermediate lists at each recursion level, requiring O(n) memory allocation per level
  • After: Yields elements one at a time using generator state, maintaining only O(1) memory for the iteration machinery regardless of comprehension size

Follow-up to:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant