Problem
The downgrade CI tests stall for 6+ hours and get cancelled. This has been happening consistently on recent runs.
Root Cause Analysis
The issue is caused by Zygote v0.7.0, which the downgrade test pins to as the minimum compatible version (compat says Zygote = "0.7").
Evidence
| Test Environment |
Julia |
Zygote |
Result |
| LTS Normal |
1.10.10 |
0.7.10 |
✅ Completes in ~25 min |
| Downgrade |
1.10.10 |
0.7.0 |
❌ Hangs for 6+ hours |
Both tests use the same:
- Julia version (1.10.10)
- GaussianMixtures version (0.3.13)
- Test code
The only significant difference is Zygote version.
Where it hangs
The hang occurs during MOE tests in AD_compatibility.jl when creating GMM models with 900 data points. The last log output before the hang:
Initializing GMM, 2 Gaussians diag covariance 3 dimensions using 900 data points
K-means converged with 8 iterations (objv = 135261.3616578743)
K-means with 900 data points using 8 iterations
112.5 data points per parameter
Then nothing for 6 hours until timeout.
The GMM constructor doesn't use Zygote directly, but there's some problematic interaction in the v0.7.0 test environment that causes the EM algorithm to hang after K-means initialization.
Potential Fixes
- Bump Zygote compat from
0.7 to 0.7.3 or higher
- Reduce test sample size in AD_compatibility.jl MOE tests from n=1000 to something smaller
- Reduce
nInit parameter in MOE extension from 50 to a smaller value
Workaround
Downgrade tests have been temporarily disabled in PR #546.
Problem
The downgrade CI tests stall for 6+ hours and get cancelled. This has been happening consistently on recent runs.
Root Cause Analysis
The issue is caused by Zygote v0.7.0, which the downgrade test pins to as the minimum compatible version (compat says
Zygote = "0.7").Evidence
Both tests use the same:
The only significant difference is Zygote version.
Where it hangs
The hang occurs during MOE tests in
AD_compatibility.jlwhen creating GMM models with 900 data points. The last log output before the hang:Then nothing for 6 hours until timeout.
The GMM constructor doesn't use Zygote directly, but there's some problematic interaction in the v0.7.0 test environment that causes the EM algorithm to hang after K-means initialization.
Potential Fixes
0.7to0.7.3or highernInitparameter in MOE extension from 50 to a smaller valueWorkaround
Downgrade tests have been temporarily disabled in PR #546.