Skip to content

Conversation

@eurunuela
Copy link
Collaborator

Summary

  • Add external regressor correlation data to the metrics file (component_table TSV)
  • When tedana generates confound_correlations.svg, the correlation matrix is now saved

Changes Made

  • Add compute_external_regressor_correlations() function to tedana/metrics/external.py
  • Modify plot_heatmap() in static_figures.py to accept precomputed correlations
  • Update tedana_workflow() to compute correlations, add to component_table, and re-save
  • Add metadata for new correlation columns in get_metadata()
  • Add unit tests for the new correlation function

Use Case

This enables Rica (and other tools) to display interactive external regressor correlation heatmaps without needing to recompute the correlations from external regressors files.

Output Format

New columns are added to the metrics TSV file with names like:

  • external regressor correlation Mot_X
  • external regressor correlation Mot_Y
  • etc.

Test plan

  • Unit tests added for compute_external_regressor_correlations()
  • All existing tests pass
  • flake8 linting passes

Closes #1294

🤖 Generated with Claude Code

When tedana generates `confound_correlations.svg` for external regressors,
the correlation matrix between external regressors and ICA components is
now saved to the metrics file (component_table TSV) alongside the figure.

Changes:
- Add `compute_external_regressor_correlations()` function to `metrics/external.py`
  for computing Pearson correlations between external regressors and ICA components
- Modify `plot_heatmap()` in `static_figures.py` to accept precomputed correlations
  instead of computing them internally (cleaner separation of concerns)
- Update `tedana_workflow()` to compute correlations, add them to component_table
  with column names like `external regressor correlation <regressor_name>`,
  and re-save the metrics file
- Add metadata for the new correlation columns in `get_metadata()`
- Add unit tests for the new correlation function

This enables Rica (and other tools) to display interactive external regressor
correlation heatmaps without needing to recompute the correlations.

Closes ME-ICA#1294

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@eurunuela eurunuela marked this pull request as ready for review January 8, 2026 17:26
@eurunuela eurunuela requested review from Copilot and tsalo January 8, 2026 17:26
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds external regressor correlation data to the component metrics TSV file, enabling tools like Rica to display interactive correlation heatmaps without needing to recompute correlations from raw regressor files.

Key changes:

  • New function compute_external_regressor_correlations() in tedana/metrics/external.py to compute Pearson correlations between external regressors and ICA components
  • Refactored plot_heatmap() in static_figures.py to accept precomputed correlations instead of computing them internally
  • Updated workflow to compute correlations, add them to component_table as new columns, and re-save the metrics file with updated metadata

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tedana/workflows/tedana.py Integrates correlation computation into workflow, adds computed correlations to component_table, and re-saves metrics files
tedana/metrics/external.py Adds new compute_external_regressor_correlations() function to compute correlations using numpy's corrcoef
tedana/reporting/static_figures.py Refactors plot_heatmap() to accept precomputed correlations; removes internal _correlate_dataframes() helper function
tedana/metrics/collect.py Adds metadata generation for new correlation columns with prefix "external regressor correlation "
tedana/tests/test_external_metrics.py Adds comprehensive tests for the new correlation function including shape validation, value range checks, and input validation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

The metrics file is first saved by selector.to_files(), and then
re-saved with the external regressor correlation columns added.
When overwrite=False (the default), this caused a RuntimeError.

Temporarily set io_generator.overwrite=True before re-saving, then
restore the original setting. Also use index-based alignment when
adding correlation columns to avoid positional misalignment issues.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ations

Per tsalo's suggestion, refactor the external regressor correlation saving
to use the existing add_df_to_file method and a new add_dict_to_file method
instead of the temporary overwrite workaround.

Changes:
- Add add_dict_to_file method to OutputGenerator class for merging dicts
  into existing JSON files
- Update workflow to use add_df_to_file for TSV and add_dict_to_file for JSON
- Add reset_index(drop=True) to ensure proper DataFrame alignment
- Add unit test for the new add_dict_to_file method

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Taylor Salo <tsalo90@gmail.com>
@codecov
Copy link

codecov bot commented Jan 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.69%. Comparing base (d0c71ef) to head (9363b7c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1297   +/-   ##
=======================================
  Coverage   89.68%   89.69%           
=======================================
  Files          29       29           
  Lines        4497     4530   +33     
  Branches      746      751    +5     
=======================================
+ Hits         4033     4063   +30     
- Misses        306      316   +10     
+ Partials      158      151    -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tsalo
Copy link
Member

tsalo commented Jan 8, 2026

The PR looks pretty good to me. I think you just need to resolve the conflicts.

@eurunuela
Copy link
Collaborator Author

That should do it. I thought my local main was up to date with upstream but apparently not. I will wait for the tests to pass and will merge after that, considering this is a fairly simple PR.

Copy link
Member

@tsalo tsalo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good to me! I don't think we're retaining artifacts on CircleCI anymore so I can't check the integration test outputs. Have you tried running it locally?

@eurunuela eurunuela merged commit 9bf1335 into ME-ICA:main Jan 8, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Save external regressor correlation data alongside SVG figure

3 participants