-
Notifications
You must be signed in to change notification settings - Fork 100
Add external regressor correlation data to metrics file #1297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add external regressor correlation data to metrics file #1297
Conversation
When tedana generates `confound_correlations.svg` for external regressors, the correlation matrix between external regressors and ICA components is now saved to the metrics file (component_table TSV) alongside the figure. Changes: - Add `compute_external_regressor_correlations()` function to `metrics/external.py` for computing Pearson correlations between external regressors and ICA components - Modify `plot_heatmap()` in `static_figures.py` to accept precomputed correlations instead of computing them internally (cleaner separation of concerns) - Update `tedana_workflow()` to compute correlations, add them to component_table with column names like `external regressor correlation <regressor_name>`, and re-save the metrics file - Add metadata for the new correlation columns in `get_metadata()` - Add unit tests for the new correlation function This enables Rica (and other tools) to display interactive external regressor correlation heatmaps without needing to recompute the correlations. Closes ME-ICA#1294 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds external regressor correlation data to the component metrics TSV file, enabling tools like Rica to display interactive correlation heatmaps without needing to recompute correlations from raw regressor files.
Key changes:
- New function
compute_external_regressor_correlations()intedana/metrics/external.pyto compute Pearson correlations between external regressors and ICA components - Refactored
plot_heatmap()instatic_figures.pyto accept precomputed correlations instead of computing them internally - Updated workflow to compute correlations, add them to component_table as new columns, and re-save the metrics file with updated metadata
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tedana/workflows/tedana.py |
Integrates correlation computation into workflow, adds computed correlations to component_table, and re-saves metrics files |
tedana/metrics/external.py |
Adds new compute_external_regressor_correlations() function to compute correlations using numpy's corrcoef |
tedana/reporting/static_figures.py |
Refactors plot_heatmap() to accept precomputed correlations; removes internal _correlate_dataframes() helper function |
tedana/metrics/collect.py |
Adds metadata generation for new correlation columns with prefix "external regressor correlation " |
tedana/tests/test_external_metrics.py |
Adds comprehensive tests for the new correlation function including shape validation, value range checks, and input validation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The metrics file is first saved by selector.to_files(), and then re-saved with the external regressor correlation columns added. When overwrite=False (the default), this caused a RuntimeError. Temporarily set io_generator.overwrite=True before re-saving, then restore the original setting. Also use index-based alignment when adding correlation columns to avoid positional misalignment issues. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ations Per tsalo's suggestion, refactor the external regressor correlation saving to use the existing add_df_to_file method and a new add_dict_to_file method instead of the temporary overwrite workaround. Changes: - Add add_dict_to_file method to OutputGenerator class for merging dicts into existing JSON files - Update workflow to use add_df_to_file for TSV and add_dict_to_file for JSON - Add reset_index(drop=True) to ensure proper DataFrame alignment - Add unit test for the new add_dict_to_file method Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Taylor Salo <tsalo90@gmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1297 +/- ##
=======================================
Coverage 89.68% 89.69%
=======================================
Files 29 29
Lines 4497 4530 +33
Branches 746 751 +5
=======================================
+ Hits 4033 4063 +30
- Misses 306 316 +10
+ Partials 158 151 -7 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
The PR looks pretty good to me. I think you just need to resolve the conflicts. |
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: milton.camachocamach <milton.camachocamach@ucalgary.ca>
…al-regressor-correlations
|
That should do it. I thought my local main was up to date with upstream but apparently not. I will wait for the tests to pass and will merge after that, considering this is a fairly simple PR. |
tsalo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes look good to me! I don't think we're retaining artifacts on CircleCI anymore so I can't check the integration test outputs. Have you tried running it locally?
Summary
confound_correlations.svg, the correlation matrix is now savedChanges Made
compute_external_regressor_correlations()function totedana/metrics/external.pyplot_heatmap()instatic_figures.pyto accept precomputed correlationstedana_workflow()to compute correlations, add to component_table, and re-saveget_metadata()Use Case
This enables Rica (and other tools) to display interactive external regressor correlation heatmaps without needing to recompute the correlations from external regressors files.
Output Format
New columns are added to the metrics TSV file with names like:
external regressor correlation Mot_Xexternal regressor correlation Mot_YTest plan
compute_external_regressor_correlations()Closes #1294
🤖 Generated with Claude Code