handle multi schemaValidation for one request

[shahinshirvani]

I have an openapi.json file which includes a multi-schema response for a request like this:

"responses": {
          "200": {
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/CountResultDto"
                },
                "examples": {
                  "example-1": {
                    "summary": "Status 200.",
                    "description": "GET `/authorization/count?userIdIn=jonny1,jonny2`",
                    "value": {
                      "count": 2
                    }
                  }
                }
              }
            },
            "description": "Request successful."
          },
          "400": {
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/ExceptionDto"
                }
              }
            },
            "description": "Returned if some of the query parameters are invalid, for example if a `sortOrder`\r\nparameter is supplied, but no `sortBy` is specified. See the\r\n[Introduction](https://docs.camunda.org/manual/latest/reference/rest/overview/#error-handling)\r\nfor the error response format."
          }
        }

I can get the first schema in the post-response to a request from the postman when I use this:

      {
        "openApiOperation": "*::/*",
        "schemaValidation": {
          "enabled": true
        }
      }

How can you get all of them? For example, in this scenario, get response 200 and 400 schema validation

[thim81]

hi @shahinshirvani

Let me do some trials, since If I'm not mistaken it would be possible.
I'll keep you posted.

[shahinshirvani]

@thim81 I appreciate your time and will wait for your response

[shahinshirvani]

Hi @thim81
Do you have any update about this issue?

[thim81]

hi @shahinshirvani

No update yet, I have been busy with getting a new OSS tool for inspecting HTTP requests out-of-the-door.
That should be done this week, which will give me more time to look into your reported Portman issue.

[thim81]

hi @shahinshirvani

What you're looking for is commonly referred to as variation testing—testing multiple expected outcomes (e.g., 200 and 400 responses) for the same request.

Since Postman executes a single request per run, the Portman way to test variations is by defining test variations that duplicate the request and apply specific transformations or expectations.

Have a look at the "Variation test" example.

In your case, you'd create one variation for the valid request (expecting 200 with schema validation), and another variation for an invalid request (expecting 400, also with schema validation). Portman allows you to specify a different expected status code and schema validation per variation.

Additionally, Portman also supports fuzz testing. This means it can automatically generate request variations based on your OpenAPI spec to test how your API behaves under unexpected or edge-case input. Fuzz testing is a great way to identify unhandled errors, schema violations.

[shahinshirvani]

@thim81 Portman is able to define test variation per requests as I understand but is Portman able to get all schema variation from open-api and duplicate requests per schema that exist in the open-api file?

[thim81]

hi @shahinshirvani

That is exactly what the objective is with the "variation" tests. In the variation, you can define what status (200, 404, ..) & content-type what is expected.

[shahinshirvani]

Hi @thim81
I know that we can define what status and content type are expected in the variation. The problem is that my open-API file already defines those parameters. So I need an option like this:

"schemaValidation": {
          "enabled": true
  }

So I can get all schema validations that are defined in the open API file.
Create 3 requests in Postman if I have 3 schema validations (e.g., status 200, 400, 500).
Assign each schema validation for each request.
I do not want to define new schema validation in Portman.

[shahinshirvani]

Hi @thim81
Do you have any update about my question?
I can explain it in another way if you want

[thim81]

hi @shahinshirvani

We have just merged a PR, that will allow the variations to generate and/or define the expected schema.

Have a look at the example: https://github.com/apideck-libraries/portman/tree/main/examples/testsuite-variation-content-types#combined-openapirequest-and-openapiresponse-example

It will be part of the upcoming release of Portman.