Swagger API docs generation: Error: Max. depth level of JSON reached at position

[MiTschMR]

Describe the issue
I am facing an issue that when having a really large swagger.json (15+ thousand lines, size was 7.538 MB), the json stringify or parse function will fail with the error message Error: Max. depth level of JSON reached at position xxx.

To Reproduce
Steps to reproduce the behavior:

1. Have a really large swagger.json
2. Let docfx parse it
3. See the error message output

Expected behavior
docfx is able to parse the swagger.json without issues.

Context (please complete the following information):

• OS: Windows 11 (Build 22631.4460)
• Docfx version: 2.77.0

Additional context
A swagger.json with around 10k lines doesn't fail. The issue seems to be caused by an underlying JS library that has the limit, but I am not a specialist there to be able to tell what exactly is going on. I am not able to share the swagger.json due to NDA. Full stacktrace:

<path to swagger.json>: error ApplyTemplatePreprocessorError: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of JSON reached at position 889036
   at parse RestApi.common.js:297:22
   at addDefinition (definition, definitions) RestApi.common.js:297:22
   at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
   at forEach RestApi.common.js:88:72
   at forEach RestApi.common.js:88:17
   at forEach RestApi.common.js:87:13
   at RestApi.common.js:86:9
DocumentException: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of
JSON reached at position 889036
   at parse RestApi.common.js:297:22
   at addDefinition (definition, definitions) RestApi.common.js:297:22
   at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
   at forEach RestApi.common.js:88:72
   at forEach RestApi.common.js:88:17
   at forEach RestApi.common.js:87:13
   at RestApi.common.js:86:9
     DocumentException: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth
     level of JSON reached at position 889036
        at parse RestApi.common.js:297:22
        at addDefinition (definition, definitions) RestApi.common.js:297:22
        at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
        at forEach RestApi.common.js:88:72
        at forEach RestApi.common.js:88:17
        at forEach RestApi.common.js:87:13
        at RestApi.common.js:86:9
          InvalidPreprocessorException: Error running Transform function inside template preprocessor: Error: Max. depth level of JSON reached at position 889036
             at parse RestApi.common.js:297:22
             at addDefinition (definition, definitions) RestApi.common.js:297:22
             at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
             at forEach RestApi.common.js:88:72
             at forEach RestApi.common.js:88:17
             at forEach RestApi.common.js:87:13
             at RestApi.common.js:86:9
            at object TransformModel(object model) in PreprocessorWithResourcePool.cs:81
            at object TransformModel(object model) in Template.cs:95
            at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:98
       at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:115
       at void <ProcessCore>b__0(InternalManifestItem item) in TemplateProcessor.cs:194
       at void <RunAll>b__0(TElement s) in DocumentExceptionExtensions.cs:80
  at void RunAll<TElement>(IEnumerable<TElement> elements, Action<TElement> action, int parallelism) in DocumentExceptionExtensions.cs:89
  at void RunAll<TElement>(IReadOnlyList<TElement> elements, Action<TElement> action, int parallelism) in DocumentExceptionExtensions.cs:60
  at List<ManifestItem> ProcessCore(List<InternalManifestItem> items, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:189
  at List<ManifestItem> Process(List<InternalManifestItem> manifest, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:88
  at List<ManifestItem> ProcessTemplate() in ManifestProcessor.cs:182
  at void Process() in ManifestProcessor.cs:53
  at void Handle(List<HostService> hostServices, int maxParallelism) in LinkPhaseHandler.cs:32
  at Manifest Build(DocumentBuildParameters parameters, IMarkdownService markdownService) in SingleDocumentBuilder.cs:67
  at void Build(IList<DocumentBuildParameters> parameters, string outputDirectory) in DocumentBuilder.cs:122
  at void BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, string baseDirectory, string outputDirectory, string templateDirectory) in DocumentBuilderWrapper.cs:42
  at string Exec(BuildJsonConfig config, BuildOptions options, string configDirectory, string outputDirectory) in RunBuild.cs:39
  at void <Execute>b__0() in DefaultCommand.cs:51
  at int Run(LogOptions options, Action run) in CommandHelper.cs:48
  at int Execute(CommandContext context, Options options) in DefaultCommand.cs:31
  at Task<int> Execute(CommandContext context, CommandSettings settings) in CommandOfT.cs:40
  at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext context, ITypeResolver resolver, IConfiguration configuration) in CommandExecutor.cs:166

[filzrev]

Reported error seems to be reported by Jint JavaScript interpreter.

• https://github.com/sebastienros/jint/blob/a506929f99398c31cf734c9ba9bcf4fea108355e/Jint/Native/Json/JsonParser.cs#L761
• https://github.com/sebastienros/jint/blob/a506929f99398c31cf734c9ba9bcf4fea108355e/Jint/Options.cs#L457-L467

By default MaxParseDepth is configured as 64.
If this value is not sufficient for specific use case.
It need to increase this option value.

Is it able to provide max depth information?

[MiTschMR]

From what I can see in the raw model json the maximum depth is 21 (42 spaces) when it fails in one occasion, another time it is 15 (30 spaces). The parsing stops in the middle of the line within the spaces, it doesn't even make it to the properties, like this:

                              *fails here on this space character*        "PvHarm": {
                                        "format": "double",
                                        "type": "number"
                                      },

[filzrev]

I've created minimum projects that can reproduce problems.

Is it able to test required maxDepth with your swagger.json file?
And if possible, provide minimum swagger code fragment that can reproduce the problem.

Program.cs

using Jint;

const int maxDepth = 64;
var data = File.ReadAllText(@"C:\Temp\swagger.json");

var engine = new Engine(options =>
{
    options.Json.MaxParseDepth = maxDepth;
})
.SetValue("data", data);

var text = "return JSON.stringify(JSON.parse(data))";

var result = engine.Evaluate(text).AsString();
Console.WriteLine("Parse operation is successfully completed.");

Issue_10393.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Jint" Version="4.1.0" />
  </ItemGroup>

</Project>

[MiTschMR]

I will try to do so when I am back, currently out of office. Thanks for the repro details!

[MiTschMR]

Alright, the issue does not occur with the provided Program.cs. However, using the same swagger.json causes the exception in docfx:

[...]
Creating output...
Searching built-in plugins in directory C:\Program Files\docfx\...
7 plug-in(s) loaded.
Building 5 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>ValidateConceptualDocumentMetadata)...
Building 1 file(s) in RestApiDocumentProcessor(BuildRestApiDocument=>ValidateRestApiDocumentMetadata=>ApplyOverwriteDocumentForRestApi)...
Building 2 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
Building 5 file(s) in TocDocumentProcessor(BuildTocDocument)...
Building 1495 file(s) in ManagedReferenceDocumentProcessor(BuildManagedReferenceDocument=>SplitClassPageToMemberLevel=>ValidateManagedReferenceDocumentMetadata=>ApplyOverwriteDocumentForMref=>FillReferenceInformation)...
Applying templates to 1508 model(s)...
<path to swagger.json>: error ApplyTemplatePreprocessorError: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of JSON reached at position 1166075
   at parse RestApi.common.js:297:22
   at addDefinition (definition, definitions) RestApi.common.js:297:22
   at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
   at forEach RestApi.common.js:88:72
   at forEach RestApi.common.js:88:17
   at forEach RestApi.common.js:87:13
   at RestApi.common.js:86:9
DocumentException: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of JSON
reached at position 1166075
   at parse RestApi.common.js:297:22
   at addDefinition (definition, definitions) RestApi.common.js:297:22
   at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
   at forEach RestApi.common.js:88:72
   at forEach RestApi.common.js:88:17
   at forEach RestApi.common.js:87:13
   at RestApi.common.js:86:9
     DocumentException: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of JSON
     reached at position 1166075
        at parse RestApi.common.js:297:22
        at addDefinition (definition, definitions) RestApi.common.js:297:22
        at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
        at forEach RestApi.common.js:88:72
        at forEach RestApi.common.js:88:17
        at forEach RestApi.common.js:87:13
        at RestApi.common.js:86:9
          InvalidPreprocessorException: Error running Transform function inside template preprocessor: Error: Max. depth level of JSON reached at position 1166075
             at parse RestApi.common.js:297:22
             at addDefinition (definition, definitions) RestApi.common.js:297:22
             at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
             at forEach RestApi.common.js:88:72
             at forEach RestApi.common.js:88:17
             at forEach RestApi.common.js:87:13
             at RestApi.common.js:86:9
            at object TransformModel(object model) in PreprocessorWithResourcePool.cs:72
            at object TransformModel(object model) in Template.cs:93
            at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:97
       at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:114
       at void <ProcessCore>b__0(InternalManifestItem item) in TemplateProcessor.cs:190
       at void <RunAll>b__0(TElement s) in DocumentExceptionExtensions.cs:95
  at void RunAll<TElement>(IEnumerable<TElement> elements, Action<TElement> action, int parallelism, CancellationToken cancellationToken) in DocumentExceptionExtensions.cs:105
  at void RunAll<TElement>(IReadOnlyList<TElement> elements, Action<TElement> action, int parallelism, CancellationToken cancellationToken) in DocumentExceptionExtensions.cs:68
  at List<ManifestItem> ProcessCore(List<InternalManifestItem> items, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:185
  at List<ManifestItem> Process(List<InternalManifestItem> manifest, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:82
  at List<ManifestItem> ProcessTemplate() in ManifestProcessor.cs:186
  at void Process() in ManifestProcessor.cs:53
  at void Handle(List<HostService> hostServices, int maxParallelism) in LinkPhaseHandler.cs:32
  at Manifest Build(DocumentBuildParameters parameters, IMarkdownService markdownService, CancellationToken cancellationToken) in SingleDocumentBuilder.cs:71
  at void Build(IList<DocumentBuildParameters> parameters, string outputDirectory, CancellationToken cancellationToken) in DocumentBuilder.cs:124
  at void BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, string baseDirectory, string outputDirectory, string templateDirectory, CancellationToken cancellationToken) in DocumentBuilderWrapper.cs:42
  at string Exec(BuildJsonConfig config, BuildOptions options, string configDirectory, string outputDirectory, CancellationToken cancellationToken) in RunBuild.cs:39
  at void <Execute>b__0() in DefaultCommand.cs:51
  at int Run(LogOptions options, Action run) in CommandHelper.cs:48
  at int Execute(CommandContext context, Options options) in DefaultCommand.cs:31
  at Task<int> Execute(CommandContext context, CommandSettings settings) in CommandOfT.cs:40
  at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext context, ITypeResolver resolver, IConfiguration configuration) in CommandExecutor.cs:166

It again happens randomly in the line, not at a certain property. On another occasion it was in the middle of a property name. To me it doesn't make any sense why it would fail on one occasion and not on the other. I am not able to provide a minimal reproduction swagger.json until at earliest in a week, again out of office, though it will most likely not be easy to make one with just dummy classes.