Skip to content

Downgrade tests stall for 6+ hours due to Zygote v0.7.0 #545

Description

@ChrisRackauckas-Claude

Problem

The downgrade CI tests stall for 6+ hours and get cancelled. This has been happening consistently on recent runs.

Root Cause Analysis

The issue is caused by Zygote v0.7.0, which the downgrade test pins to as the minimum compatible version (compat says Zygote = "0.7").

Evidence

Test Environment Julia Zygote Result
LTS Normal 1.10.10 0.7.10 ✅ Completes in ~25 min
Downgrade 1.10.10 0.7.0 ❌ Hangs for 6+ hours

Both tests use the same:

  • Julia version (1.10.10)
  • GaussianMixtures version (0.3.13)
  • Test code

The only significant difference is Zygote version.

Where it hangs

The hang occurs during MOE tests in AD_compatibility.jl when creating GMM models with 900 data points. The last log output before the hang:

Initializing GMM, 2 Gaussians diag covariance 3 dimensions using 900 data points
K-means converged with 8 iterations (objv = 135261.3616578743)
K-means with 900 data points using 8 iterations
112.5 data points per parameter

Then nothing for 6 hours until timeout.

The GMM constructor doesn't use Zygote directly, but there's some problematic interaction in the v0.7.0 test environment that causes the EM algorithm to hang after K-means initialization.

Potential Fixes

  1. Bump Zygote compat from 0.7 to 0.7.3 or higher
  2. Reduce test sample size in AD_compatibility.jl MOE tests from n=1000 to something smaller
  3. Reduce nInit parameter in MOE extension from 50 to a smaller value

Workaround

Downgrade tests have been temporarily disabled in PR #546.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions