As a user, I want my long NYCDEP HEFS evaluation to succeed when running using the prospective 7.4 RC

[HankHerr-NOAA]

See #774. Upon deploying the RC to staging and migrating the db, the example NYCDEP evaluation failed. I confirmed that it also fails using a standalone pointed to a Postgres server. The declaration of that evaluation is below. The files are being moved now; they are big.

EDIT: Exception that occurred is the following:

2026-04-03T13:32:59.659+0000  [pool-24-thread-1] INFO EvaluationService - The evaluation failed with the following stack trace: 
wres.pipeline.InternalWresException: Could not complete project execution
	at wres.pipeline.Evaluator.evaluate(Evaluator.java:408)
	at wres.pipeline.Evaluator.evaluate(Evaluator.java:181)
	at wres.Functions.evaluate(Functions.java:131)
	at wres.server.EvaluationService.lambda$startEvaluation$2(EvaluationService.java:696)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: wres.pipeline.WresProcessingException: Encountered an error while processing evaluation '5ZkeZ89gYnLN4wW-S4nNCIf9mCs': 
	at wres.pipeline.Evaluator.evaluate(Evaluator.java:855)
	at wres.pipeline.Evaluator.evaluate(Evaluator.java:379)
	... 7 common frames omitted
Caused by: java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot build a climatology without a measurement unit. 
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
	... 3 common frames omitted
Caused by: java.lang.IllegalArgumentException: Cannot build a climatology without a measurement unit. 
	at wres.datamodel.types.Climatology.validate(Climatology.java:398)
	at wres.datamodel.types.Climatology.<init>(Climatology.java:363)
	at wres.datamodel.types.Climatology$Builder.build(Climatology.java:303)
	at wres.pipeline.pooling.PoolsGenerator.lambda$createClimatology$11(PoolsGenerator.java:1469)
	at wres.io.retrieving.CachingSupplier.get(CachingSupplier.java:50)
	at wres.pipeline.pooling.PoolSupplier.createPoolFromPairs(PoolSupplier.java:939)
	at wres.pipeline.pooling.PoolSupplier.createPool(PoolSupplier.java:892)
	at wres.pipeline.pooling.PoolSupplier.createPool(PoolSupplier.java:266)
	at wres.pipeline.pooling.PoolSupplier.get(PoolSupplier.java:218)
	at wres.pipeline.pooling.PoolSupplier.get(PoolSupplier.java:84)
	at wres.pipeline.pooling.PoolFactory$SupplierWithPoolRequest.get(PoolFactory.java:2894)
	at wres.pipeline.pooling.PoolProcessor.get(PoolProcessor.java:144)
	at wres.pipeline.pooling.PoolProcessor.get(PoolProcessor.java:61)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	... 3 common frames omitted

Thanks,

Hank

=======================

label: HEFS_EnsPost Streamflow
observed:
  label: OBS Streamflow
  sources: DWNN6DEL_QME.xml
  variable: QME
  feature_authority: nws lid
  type: observations
predicted:
  label: HEFS_EnsPost Streamflow
  sources: DWNN6TIF_HEFS_EnsPost_SQIN.tgz
  variable: SQIN
  type: ensemble forecasts
baseline:
  label: POSTPROCESSED Streamflow
  sources: DWNN6TOT_POSTPROCESSED_SQIN.tgz
  variable: SQIN
  type: ensemble forecasts
  separate_metrics: true
features:
  - {observed: DWNN6DEL, predicted: DWNN6TIF, baseline: DWNN6TOT}
unit: cms
lead_times:
  minimum: 18
  maximum: 720
  unit: hours
lead_time_pools:
  period: 24
  frequency: 24
  unit: hours
time_scale:
  function: mean
  period: 24
  unit: hours
pair_frequency:
  period: 24
  unit: hours
cross_pair: exact
probability_thresholds: [0.01,0.1,0.5,0.9,0.95,0.99,0.995,0.999]
minimum_sample_size: 30
season:
  minimum_day: 1
  minimum_month: 1
  maximum_day: 31
  maximum_month: 12
metrics:
  - box plot of errors by forecast value
  - continuous ranked probability score
  - relative operating characteristic score
  - brier score
  - pearson correlation coefficient
  - root mean square error
  - mean error
  - bias fraction
  - reliability diagram
  - relative operating characteristic diagram
  - ensemble quantile quantile diagram
  - brier skill score
  - name: quantile quantile diagram
    thresholds: all data
  - continuous ranked probability skill score
  - sample size
  - mean absolute error
  - box plot of errors by observed value
  - name: rank histogram
    probability_thresholds:
      values: [0.01,0.1,0.5,0.9,0.95,0.99,0.995,0.999]
      apply_to: predicted
combined_graphics: true
duration_format: hours
decimal_format: '0.000'
output_formats:
  - csv2
  - format: png
    orientation: lead threshold

[HankHerr-NOAA]

@james-d-brown If you see anything obviously wrong with the declaration that could lead to the units exception, please let mek now. I'll update the description with the exception that occurs.

Hank

[HankHerr-NOAA]

The data download from the NWC VM is reporting nearly 2.5 hours to complete for the two ensemble files. Wow. That should be about a half-hour before I need to leave. I hope the upload to Google will be faster.

Hank

[HankHerr-NOAA]

@james-d-brown Might have a minor change for you. When encountering a 502, I see this log message in stdout:

2026-04-03T17:48:55.388+0000 [USGS NWIS Reading Thread 6] WARN WebClient - Retrying https://api.waterdata.usgs.gov/ogcapi/v0/collections/continuous/items?api_key=[omitted]&f=json&lang=en-US&limit=50000&monitoring_location_id=USGS-01593450&parameter_code=00060&properties=id%2Ctime_series_id%2Cmonitoring_location_id%2Cstatistic_id%2Ctime%2Cvalue%2Cunit_of_measure&skipGeometry=true&time=2026-01-01T00%3A00%3A01Z%2F2026-03-09T07%3A58%3A53Z in a bit due to HTTP status 502.

Only, the API key was not omitted (as shown above); it was included in the log message. In staging this is not critical, but once it goes to production, I believe the plan was to redact keys from all logging. Correct?

Thanks,

Hank

[HankHerr-NOAA]

The evaluation is being processed, some pools leading to stats, some not, so I think this and my previous comment confirms that the key was handled correctly. I'll rerun a variation of this test when we deploy a new RC that address the HEFS evaluation failure. That issue was reported in #781.

Thanks,

Hank

[james-d-brown]

@james-d-brown Might have a minor change for you. When encountering a 502, I see this log message in stdout:

2026-04-03T17:48:55.388+0000 [USGS NWIS Reading Thread 6] WARN WebClient - Retrying https://api.waterdata.usgs.gov/ogcapi/v0/collections/continuous/items?api_key=[omitted]&f=json&lang=en-US&limit=50000&monitoring_location_id=USGS-01593450&parameter_code=00060&properties=id%2Ctime_series_id%2Cmonitoring_location_id%2Cstatistic_id%2Ctime%2Cvalue%2Cunit_of_measure&skipGeometry=true&time=2026-01-01T00%3A00%3A01Z%2F2026-03-09T07%3A58%3A53Z in a bit due to HTTP status 502.

Only, the API key was not omitted (as shown above); it was included in the log message. In staging this is not critical, but once it goes to production, I believe the plan was to redact keys from all logging. Correct?

Thanks,

Hank

Ah, this one may be harder to fix. This is one reason you don't expose logging to users, because it may contain private info. The problem is that the WebClient accepts generic URIs. We would need to have a centralized method for redacting URIs, I suppose. Anyway, that will need a separate ticket. The stuff we redact currently is related to system property overrides and system settings, not all logging everywhere. It's quite probably we leak other info via logging, hence not providing logging to users, although we are stuck with that for now.

[james-d-brown]

@james-d-brown If you see anything obviously wrong with the declaration that could lead to the units exception, please let mek now. I'll update the description with the exception that occurs.

Hank

This isn't an exception that relates to declaration or data, else the message would say something about what to do, it's definitely a bug of some kind.

[HankHerr-NOAA]

Test data for the declaration in the description is in the VLab large data sets folder, GitHub_781.

Hank

[HankHerr-NOAA]

Ah, this one may be harder to fix. This is one reason you don't expose logging to users, because it may contain private info. The problem is that the WebClient accepts generic URIs. We would need to have a centralized method for redacting URIs, I suppose. Anyway, that will need a separate ticket. The stuff we redact currently is related to system property overrides and system settings, not all logging everywhere. It's quite probably we leak other info via logging, hence not providing logging to users, although we are stuck with that for now.

I created #783 for the redaction of the USGS API key. Thanks,

Hank

[james-d-brown]

Need to create a small reproducible example, this dataset is way to big for debugging.

[james-d-brown]

Reproduced with a single file for each set of forecasts.

Caused by: java.lang.IllegalArgumentException: Cannot build a climatology without a measurement unit.
at wres.datamodel.types.Climatology.validate(Climatology.java:398)
at wres.datamodel.types.Climatology.(Climatology.java:363)
at wres.datamodel.types.Climatology$Builder.build(Climatology.java:303)
at wres.pipeline.pooling.PoolsGenerator.lambda$createClimatology$12(PoolsGenerator.java:1493)
at wres.io.retrieving.CachingSupplier.get(CachingSupplier.java:50)
at wres.pipeline.pooling.PoolSupplier.createPoolFromPairs(PoolSupplier.java:939)
at wres.pipeline.pooling.PoolSupplier.createPool(PoolSupplier.java:892)
at wres.pipeline.pooling.PoolSupplier.createPool(PoolSupplier.java:266)