upload ends in timeout: Rest API failure with HTTP(S) status 500 timeout receiving response (>30 secs)

[ksand-ruv]

We use zowe cli to do automated uploads of git repository contents to our mainframe (both in pipelines and from local developer machines). In some (seemingly random) cases we are encountering errors when uploading.
This is an example of the error occuring.

Commands: zowe files upload dir-to-pds [...] [...]

success: false
from: [...]
to: [...]
error:
message:
Rest API failure with HTTP(S) status 500
receiveResponseHeader: timeout receiving response (>30 secs): TsoServerConnection(USER=[...], ASID=0x0283, QID=0x0010000c)

stack:
Error: Rest API failure with HTTP(S) status 500
receiveResponseHeader: timeout receiving response (>30 secs): TsoServerConnection(USER=[...], ASID=0x0283, QID=0x0010000c)
at ZosmfRestClient.populateError (/usr/local/lib/node_modules/@zowe/cli/node_modules/@zowe/imperative/lib/rest/src/client/AbstractRestClient.js:729:16)
at requestEnd (/usr/local/lib/node_modules/@zowe/cli/node_modules/@zowe/imperative/lib/rest/src/client/AbstractRestClient.js:631:35)
at ZosmfRestClient.onEnd (/usr/local/lib/node_modules/@zowe/cli/node_modules/@zowe/imperative/lib/rest/src/client/AbstractRestClient.js:646:13)
at IncomingMessage. (/usr/local/lib/node_modules/@zowe/cli/node_modules/@zowe/imperative/lib/rest/src/client/AbstractRestClient.js:556:18)
at IncomingMessage.emit (node:events:531:35)
at endReadableNT (node:internal/streams/readable:1698:12)
at process.processTicksAndRejections (node:internal/process/task_queues:90:21)
file_to_upload: 768
success: 26
error: 1
skipped: 741
Rest API failure with HTTP(S) status 500
receiveResponseHeader: timeout receiving response (>30 secs): TsoServerConnection(USER=[...], ASID=0x0283, QID=0x0010000c)
Unable to perform this operation due to the following problem.
receiveResponseHeader: timeout receiving response (>30 secs): TsoServerConnection(USER=[...], ASID=0x0283, QID=0x0010000c)

We are uploading to a previously created dataset (command used: zowe files create pds "[...]" --dsntype LIBRARY --recfm FB --lrecl 80 --asu CYL --ps 10 --ss 10 --blksize 32720)

Observations:

• Increased timeout parameter (up to 10 minutes max) has no effect
• Once an error scenario occurs, always the same file fails (also if uploaded as a single file)
• Upload may work again when making (seemingly random) changes to the file in question
• Upload fails from all locations once errourous state is reached

[github-actions]

Thank you for creating a bug report.
We will investigate the bug and evaluate its impact on the product.
If you haven't already, please ensure you have provided steps to reproduce the bug and as much context as possible.

[traeok]

Hi @ksand-ruv ,

This reminds me of issue #2662 where if a data set upload exceeds 809 lines, z/OSMF indefinitely hangs and an error is eventually returned after the TSO session times out. Can you verify how many lines you are uploading at once to the failed/skipped data sets?

https://ideas.ibm.com/ideas/ZOS-I-4736 This issue tracks the behavior exhibited by z/OSMF APIs. I'm curious to know if the issue details align with the behavior you're seeing?

Thanks!

[ksand-ruv]

Hi @traeok
We encountered the error only with data sets exceeding 809 lines. But we also have dataset exceeding these limits that work fine.

We looked over the issures and the idea you mentioned an we don't believe they apply to our problems. The examples there have multi-byte encouding (japanese/chinese symbols) but our data sets have single byte encouding (we use the latin alphabet).