Skip to content

Cohort evaluation rubric metrics#21

Open
azimov wants to merge 66 commits intodevelopfrom
cohort-evaluation-metrics
Open

Cohort evaluation rubric metrics#21
azimov wants to merge 66 commits intodevelopfrom
cohort-evaluation-metrics

Conversation

@azimov
Copy link
Collaborator

@azimov azimov commented Mar 18, 2026

Rulesets for cohort evaluation.

Done:

  • basic skills and API for building cohorts with coding agents
  • rubric evaluation framework (produce a report about cohorts)

TODO:

  • Integrate ibis execution and retire use of t-sql

azimov and others added 30 commits January 21, 2026 09:25
…api-tooling

# Conflicts:
#	gpt4o_cohort_builder_system_prompt.md
…ch provide llms with context in complex nested situations. Added functionality for agents to get this skill and use it
azimov and others added 25 commits February 6, 2026 08:41
# Conflicts:
#	circe/__init__.py
#	circe/api.py
#	circe/vocabulary/__init__.py
# Conflicts:
#	README.md
#	circe/__init__.py
#	circe/api.py
#	circe/chat.py
#	circe/cli.py
#	circe/cohort_builder/builder.py
#	circe/cohortdefinition/__init__.py
#	circe/cohortdefinition/cohort.py
#	circe/cohortdefinition/criteria.py
#	circe/cohortdefinition/printfriendly/markdown_render.py
#	circe/helper/__init__.py
#	circe/helper/cohort_modifiers.py
#	pyproject.toml
#	tests/test_cohort_expression.py
@codecov
Copy link

codecov bot commented Mar 18, 2026

Codecov Report

❌ Patch coverage is 73.57456% with 482 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.70%. Comparing base (252e68c) to head (d5a8be9).

Files with missing lines Patch % Lines
circe/cohort_builder/query_builder.py 68.11% 162 Missing ⚠️
circe/extensions/__init__.py 55.14% 109 Missing ⚠️
circe/capr/query.py 64.28% 95 Missing ⚠️
circe/evaluation/builder.py 82.29% 37 Missing ⚠️
circe/execution/api.py 48.71% 20 Missing ⚠️
circe/capr/cohort.py 88.03% 14 Missing ⚠️
circe/evaluation/markdown_render.py 76.00% 12 Missing ⚠️
circe/execution/evaluation.py 80.00% 7 Missing ⚠️
circe/capr/templates.py 84.61% 6 Missing ⚠️
circe/capr/window.py 87.50% 5 Missing ⚠️
... and 6 more
Additional details and impacted files
@@             Coverage Diff             @@
##           develop      #21      +/-   ##
===========================================
- Coverage    85.37%   83.70%   -1.68%     
===========================================
  Files          167      185      +18     
  Lines        12386    15098    +2712     
===========================================
+ Hits         10575    12638    +2063     
- Misses        1811     2460     +649     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant