Stale GCS logs

[Json-Andriopoulos]

Contact Details [Optional]

No response

System Information

No response

What happened?

Step execution logs are prematurely cached when refreshed during execution.

Reproduction steps

Use a pipeline over the GCP stack and a step that generates mutliple log records for some time. During execution, attempt to refresh the logs . This will prematurely cache the logs - even refreshing post completion does not show the entirety of the logs.

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct

[alekseevpavel04]

I'd like to take this one.

The caching looks like fsspec's directory-listing cache rather than anything in
the log store. Since #4974 the server keeps one artifact store instance per
component in an LRU (instantiate_artifact_store →
zen_server/artifact_store_cache.py), so a single gcsfs.GCSFileSystem is
shared across requests. dircache is only invalidated by writes made through
that same instance, and the log files are written by the orchestrator in another
process — so the first log request populates the cache for the log directory and
nothing ever clears it. _stream_logs_line_by_line then lists a stale set of
files and opens each one at its cached size, which is the "prematurely cached"
part.

It also explains why refreshing after completion does not recover the rest:
ArtifactLogExporter._merge replaces the per-chunk files with a single
<ts>_merged.log, so the cached listing names files that no longer exist and
does not name the one that does.

AzureArtifactStore already constructs its filesystem with
use_listings_cache=False; GCPArtifactStore and S3ArtifactStore do not. My
plan is to pass the same flag in both, so the three stores agree — for S3 it
goes in _build_filesystem_kwargs() so the S3-compatible subclasses inherit it.

One question, since it changes the diff: would you rather bound the staleness
than remove the cache — a short cache_timeout / listings_expiry_time instead
of disabling it? Turning it off entirely also costs a listing round-trip on
client-side reads, where the cache is correct because the same process does the
writes. I will go with the Azure-consistent version unless you would prefer
otherwise.