GetFlows: unbounded list response causes HTTP 500 in large environments; ?attributes= workaround blocked by required-field validation

[samir-gandhi]

Summary

Two related bugs in DaVinciFlowsApi.GetFlows() that together make the list-flows operation unreliable for environments with many or large flows.

---

Bug 1: Unbounded list response exceeds 10 MB AWS API Gateway limit

SDK version: v0.10.1

GetFlows() fetches all flows in a single unbounded response with no way to limit the fields or page the results. In environments with many or large flows the response body exceeds the 10 MB AWS API Gateway limit. The gateway returns HTTP 500, and the SDK's retry loop retries on 500 but cannot recover because every attempt hits the same size limit.

Observed log output:

INFO HTTP status code detected, available for retry "status code"=500
ERROR Server error detected, retrying "status code"=500
INFO API call failed, backing off by calculated duration. "retry attempt"=1 error=<nil> "backoff duration"=2.033s
INFO HTTP status code detected, available for retry "status code"=500
ERROR Server error detected, retrying "status code"=500
INFO API call failed, backing off by calculated duration. "retry attempt"=2 error=<nil> "backoff duration"=4.023s

The error=<nil> is expected — the HTTP transport succeeds but the gateway returns 500 due to response size.

The underlying API supports ?attributes=id,name which limits each record to just its ID and name (~120 KB for 100 flows vs. many MB for full graph data). The SDK exposes this via .Attributes("id,name") — but this hits Bug 2.

---

Bug 2: ?attributes= response fails SDK unmarshal validation

When .Attributes("id,name,environment") is passed to limit the response, DaVinciFlowResponse.UnmarshalJSON and DaVinciFlowResponseLinks.UnmarshalJSON both enforce hardcoded required-field lists that the partial response cannot satisfy.

DaVinciFlowResponse requires: _links, environment, id, name

DaVinciFlowResponseLinks requires: environment, self, connectorInstances, connectors, flow.deploy, flow.clone, flow.enabled, version

The version link is never returned by the list endpoint regardless of what ?attributes= is passed, so it is impossible to construct a valid ?attributes= value that satisfies the SDK's unmarshal validation.

Error:

no value given for required property version when unmarshaling DaVinciFlowResponseLinks

---

Workaround in pingcli-plugin-terraformer

We bypass GetFlows() entirely with a raw HTTP call to GET /flows?attributes=id,name, then call GetFlowById per flow for full details. This keeps the list response small regardless of environment size.

// WORKAROUND: The SDK's GetFlows() unmarshals into DaVinciFlowResponse /
// DaVinciFlowResponseLinks, both of which enforce required-field validation in
// UnmarshalJSON. The list endpoint omits several fields that the SDK requires
// (e.g. the "version" link in _links) regardless of what ?attributes= is passed,
// causing an unmarshal error. Additionally, without ?attributes= the list response
// includes full flow graph data, which can exceed the 10 MB AWS API Gateway limit
// and return HTTP 500 for environments with many or large flows.
func listFlowIDs(ctx context.Context, c *Client) ([]flowID, error) {
    cfg := c.apiClient.GetConfig()
    reqURL := fmt.Sprintf("%s://%s/v1/environments/%s/flows?attributes=id,name",
        cfg.Scheme, cfg.Host, c.environmentID.String())
    // ... raw HTTP call using cfg.HTTPClient
}

---

Suggested fixes

Bug 1: Expose a way to request a minimal list response — either document that callers must use .Attributes() for large environments, or add explicit support for a lightweight list-IDs-only mode.

Bug 2: Remove _links sub-fields and version from the required-property list in DaVinciFlowResponseLinks.UnmarshalJSON. Fields that the list endpoint never returns should not be required for unmarshal to succeed. Alternatively, make the required-field validation conditional on whether an ?attributes= filter was applied.

---

Environment

• SDK version: github.com/pingidentity/pingone-go-client v0.10.1
• Reproduced against: PingOne AP region, environment with many flows
• Related workarounds already in this SDK: GetFlowById uses raw HTTP for the same reason (see WORKAROUND_RAW_HTTP.md in pingcli-plugin-terraformer)