fix: deep array merge

[lipnitsk]

deep-extend does not support array merge. There was special code added
to merge top-level arrays, but that was a shallow merge.

Use deepmerge instead of deep-extend to merge arrays also. Default merge
settings seem to work well - all tests pass.

Add a test case based on swagger-api/swagger-ui#7618

Fixes: 2f5bb86 ("Fix and test for swagger-ui #3328: swagger-api/swagger-ui#3328. Added manual logic to merge arrays after calling deepExtend within mergeDeep function")

@dballance do you mind checking this against your large spec? I wonder how well deepmerge does performance-wise and it would be nice to avoid extra complexity and just let the library do the merge. I would have tested it, but couldn't find your large spec in the tests. Reference #1150, #1190, #1209, #1217. Potential alternative if perfomance becomes an issue: https://www.npmjs.com/package/deepmerge-ts

Motivation and Context

Fixes swagger-api/swagger-ui#7618

Types of changes

• No code changes (changes to documentation, CI, metadata, etc)
• Dependency changes (any modification to dependencies in package.json)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.

[dballance]

@lipnitsk - unfortunately, I've moved on from the previous job where we had this spec, so I don't have ready access to test.

I've reached out to some former coworkers to see if they can send me a sample - but will also check my personal archive to see if I have a copy lying around.

[lipnitsk]

@dballance thanks! I also noticed that deepmerge has a clone option that defaults to true. Since we don't really care about immutability of the original object in this case we could set it to false. Note that clone will be un-deprecated in the upcoming v5 release of deepmerge.

[dballance]

I've got a copy of the json OpenAPI spec that is massive.

Gonna try to get it tested this week - and also see if I can rework it to remove identifying features so it can be added as a test in the repo here.

[dballance]

@lipnitsk Finally got some time to sit down and test this. Everything appears to be working with this spec, with a slight regression in resolution speed.

swagger-api:master resolves in ~6.8s (rough average).
lipnitsk:master resolves in ~9.8s (rough average).

So, ~44% regression in spec resolution speed. Seems fine to me given the nature of the size of the spec - just wanted to provide the data so the maintainers have it for reference.

The only comment I would have is that both swagger-api:master and lipnitsk:master produce an equivalent result from Swagger({ spec }). Is this expected? I would expect that something would change to resolve the required properties. It could very well be that this spec doesn't touch that specific case for Swagger UI / OpenAPI specs - from a quick look, it doesn't look like any properties are required in the API spec...

Spec Stats

• 5.4MB on disk
• 191k+ lines of formatted JSON
• 473 definitions
• 938 path definitions (OData!)

Verification

To verify, I ran the below test on both branches (swagger-api:master & lipnitsk:master), and then verified that:

• A diff of huge-parsed.json from both branches is equivalent
• The number of expected definitions matches what I can grok from the original spec file

Test

describe('huge spec', () => {
  let spec;
  
  beforeAll(() => {
    spec = require('./data/huge.json');
  });

  test('should resolve', async () => {
    console.time('huge');
    const first = await Swagger({ spec });
    console.timeEnd('huge');
    expect(first.spec.info.title).toEqual('something');
    fs.writeFileSync('./huge-parsed.json', JSON.stringify(first.spec, null, 2), 'utf-8');
  });
});

[lipnitsk]

Seems fine to me given the nature of the size of the spec

@dballance agreed. If performance becomes an issue it can be optimized, but your test case is probably one of the worst. Until such need arises, keeping things simple is probably best. @char0n can we merge this fix?

Thanks for testing and such a detailed response, btw!

[lipnitsk]

@char0n, can we merge this please?

[char0n]

@lipnitsk sorry for late response. Looking into this now

[char0n]

Firstly thank you for contributing and collaborating on this issue/PR.

Replacing deep-extend for deepmerge seems like an optimal solution as it handles merging arrays deeply as you've already explained. Never the less, I was also thinking we can use lodash to get the job done. This is how lodash solution for replacing deep-extend looks like:

const { mergeWith } = require('lodash/fp')

const merge = mergeWith((objValue, srcValue) => {
	if (Array.isArray(objValue)) {
		if (!Array.isArray(srcValue)) {
			return srcValue;
		}
		return objValue.concat(srcValue)
	}
});

merge(target, source); // immutable merge 

I've run this merge function against deep-extend and deepmerge test suite and it's compatible with both, except one use-case - undefined values in source objects don't override values in target object. This is probably primary reason why deepmerge exist. Having said that, I think is safer for now to go with deepmerge.

Regarding performance - it has been discussed in this PR that there is a performance degradation when using deepmerge. I haven't been able to verify this. I've run statistical performance analysis using benchmark.js model under 1% margin of error and got following results:

lodash: '1,588 ops/sec ±0.72% (674 runs sampled)',
deepmerge: '1,586 ops/sec ±0.91% (671 runs sampled)',
master: '1,576 ops/sec ±0.69% (674 runs sampled)',

for the following definition:

{
  "swagger": "2.0",
  "info": {
    "version": "1.0.0",
    "title": "Swagger Petstore",
    "description": "A sample API that uses a petstore as an example to demonstrate features in the swagger-2.0 specification",
    "termsOfService": "http://swagger.io/terms/",
    "contact": {
      "name": "Swagger API Team"
    },
    "license": {
      "name": "MIT"
    }
  },
  "host": "petstore.swagger.io",
  "basePath": "/api",
  "schemes": [
    "http"
  ],
  "consumes": [
    "application/json"
  ],
  "produces": [
    "application/json"
  ],
  "paths": {
    "/pets": {
      "get": {
        "description": "Returns all pets from the system that the user has access to",
        "operationId": "findPets",
        "produces": [
          "application/json",
          "application/xml",
          "text/xml",
          "text/html"
        ],
        "parameters": [
          {
            "name": "tags",
            "in": "query",
            "description": "tags to filter by",
            "required": false,
            "type": "array",
            "items": {
              "type": "string"
            },
            "collectionFormat": "csv"
          },
          {
            "name": "limit",
            "in": "query",
            "description": "maximum number of results to return",
            "required": false,
            "type": "integer",
            "format": "int32"
          }
        ],
        "responses": {
          "200": {
            "description": "pet response",
            "schema": {
              "type": "array",
              "items": {
                "$ref": "#/definitions/Pet"
              }
            }
          },
          "default": {
            "description": "unexpected error",
            "schema": {
              "$ref": "#/definitions/ErrorModel"
            }
          }
        }
      },
      "post": {
        "description": "Creates a new pet in the store.  Duplicates are allowed",
        "operationId": "addPet",
        "produces": [
          "application/json"
        ],
        "parameters": [
          {
            "name": "pet",
            "in": "body",
            "description": "Pet to add to the store",
            "required": true,
            "schema": {
              "$ref": "#/definitions/NewPet"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "pet response",
            "schema": {
              "$ref": "#/definitions/Pet"
            }
          },
          "default": {
            "description": "unexpected error",
            "schema": {
              "$ref": "#/definitions/ErrorModel"
            }
          }
        }
      }
    },
    "/pets/{id}": {
      "get": {
        "description": "Returns a user based on a single ID, if the user does not have access to the pet",
        "operationId": "findPetById",
        "produces": [
          "application/json",
          "application/xml",
          "text/xml",
          "text/html"
        ],
        "parameters": [
          {
            "name": "id",
            "in": "path",
            "description": "ID of pet to fetch",
            "required": true,
            "type": "integer",
            "format": "int64"
          }
        ],
        "responses": {
          "200": {
            "description": "pet response",
            "schema": {
              "$ref": "#/definitions/Pet"
            }
          },
          "default": {
            "description": "unexpected error",
            "schema": {
              "$ref": "#/definitions/ErrorModel"
            }
          }
        }
      },
      "delete": {
        "description": "deletes a single pet based on the ID supplied",
        "operationId": "deletePet",
        "parameters": [
          {
            "name": "id",
            "in": "path",
            "description": "ID of pet to delete",
            "required": true,
            "type": "integer",
            "format": "int64"
          }
        ],
        "responses": {
          "204": {
            "description": "pet deleted"
          },
          "default": {
            "description": "unexpected error",
            "schema": {
              "$ref": "#/definitions/ErrorModel"
            }
          }
        }
      }
    }
  },
  "definitions": {
    "Pet": {
      "type": "object",
      "allOf": [
        {
          "$ref": "#/definitions/NewPet"
        },
        {
          "required": [
            "id"
          ],
          "properties": {
            "id": {
              "type": "integer",
              "format": "int64"
            }
          }
        }
      ]
    },
    "NewPet": {
      "type": "object",
      "required": [
        "name"
      ],
      "properties": {
        "name": {
          "type": "string"
        },
        "tag": {
          "type": "string"
        }
      }
    },
    "ErrorModel": {
      "type": "object",
      "required": [
        "code",
        "message"
      ],
      "properties": {
        "code": {
          "type": "integer",
          "format": "int32"
        },
        "message": {
          "type": "string"
        }
      }
    }
  }
}

[char0n]

@dballance would you be able to run your perf test again using benchmark.js?

Here is a how the performance test looks like using benchmark.js. You can play with minSample value. Ideally the margin of error needs to be under 1% (in master test the margin of error is 0.69%). File can by run by

 $ node pefr.js

const Benchmark = require('benchmark');

const { default: SwaggerClient } = require('../lib/index.js');

const spec = require('./data/huge.json');

const options = {
  name: 'SwaggerClient.resolveSubtree',
  defer: true,
  // minSamples: 600,
  expectedLodash: '1,588 ops/sec ±0.72% (674 runs sampled)',
  expectedDeepmerge: '1,586 ops/sec ±0.91% (671 runs sampled)',
  master: '1,576 ops/sec ±0.69% (674 runs sampled)',
  async fn(deferred) {
    await SwaggerClient.resolveSubtree(spec, []);
    deferred.resolve();
  },
};

module.exports = options;

// we're running as a script
if (module.parent === null) {
  const bench = new Benchmark({
    ...options,
    onComplete(event) {
      console.info(String(event.target));
    },
    onError(event) {
      console.error(event);
    },
  });
  bench.run();
}

[char0n]

Looks good! Before we move forward, please have a look at my code review comments.

[char0n]

Looks good! Let's merge and release.

[swagger-bot]

🎉 This PR is included in version 3.18.1 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀