Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swagger API docs generation: Error: Max. depth level of JSON reached at position #10393

Open
MiTschMR opened this issue Nov 16, 2024 · 5 comments

Comments

@MiTschMR
Copy link

Describe the issue
I am facing an issue that when having a really large swagger.json (15+ thousand lines, size was 7.538 MB), the json stringify or parse function will fail with the error message Error: Max. depth level of JSON reached at position xxx.

To Reproduce
Steps to reproduce the behavior:

  1. Have a really large swagger.json
  2. Let docfx parse it
  3. See the error message output

Expected behavior
docfx is able to parse the swagger.json without issues.

Context (please complete the following information):

  • OS: Windows 11 (Build 22631.4460)
  • Docfx version: 2.77.0

Additional context
A swagger.json with around 10k lines doesn't fail. The issue seems to be caused by an underlying JS library that has the limit, but I am not a specialist there to be able to tell what exactly is going on. I am not able to share the swagger.json due to NDA. Full stacktrace:

<path to swagger.json>: error ApplyTemplatePreprocessorError: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of JSON reached at position 889036
   at parse RestApi.common.js:297:22
   at addDefinition (definition, definitions) RestApi.common.js:297:22
   at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
   at forEach RestApi.common.js:88:72
   at forEach RestApi.common.js:88:17
   at forEach RestApi.common.js:87:13
   at RestApi.common.js:86:9
DocumentException: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of
JSON reached at position 889036
   at parse RestApi.common.js:297:22
   at addDefinition (definition, definitions) RestApi.common.js:297:22
   at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
   at forEach RestApi.common.js:88:72
   at forEach RestApi.common.js:88:17
   at forEach RestApi.common.js:87:13
   at RestApi.common.js:86:9
     DocumentException: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth
     level of JSON reached at position 889036
        at parse RestApi.common.js:297:22
        at addDefinition (definition, definitions) RestApi.common.js:297:22
        at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
        at forEach RestApi.common.js:88:72
        at forEach RestApi.common.js:88:17
        at forEach RestApi.common.js:87:13
        at RestApi.common.js:86:9
          InvalidPreprocessorException: Error running Transform function inside template preprocessor: Error: Max. depth level of JSON reached at position 889036
             at parse RestApi.common.js:297:22
             at addDefinition (definition, definitions) RestApi.common.js:297:22
             at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
             at forEach RestApi.common.js:88:72
             at forEach RestApi.common.js:88:17
             at forEach RestApi.common.js:87:13
             at RestApi.common.js:86:9
            at object TransformModel(object model) in PreprocessorWithResourcePool.cs:81
            at object TransformModel(object model) in Template.cs:95
            at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:98
       at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:115
       at void <ProcessCore>b__0(InternalManifestItem item) in TemplateProcessor.cs:194
       at void <RunAll>b__0(TElement s) in DocumentExceptionExtensions.cs:80
  at void RunAll<TElement>(IEnumerable<TElement> elements, Action<TElement> action, int parallelism) in DocumentExceptionExtensions.cs:89
  at void RunAll<TElement>(IReadOnlyList<TElement> elements, Action<TElement> action, int parallelism) in DocumentExceptionExtensions.cs:60
  at List<ManifestItem> ProcessCore(List<InternalManifestItem> items, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:189
  at List<ManifestItem> Process(List<InternalManifestItem> manifest, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:88
  at List<ManifestItem> ProcessTemplate() in ManifestProcessor.cs:182
  at void Process() in ManifestProcessor.cs:53
  at void Handle(List<HostService> hostServices, int maxParallelism) in LinkPhaseHandler.cs:32
  at Manifest Build(DocumentBuildParameters parameters, IMarkdownService markdownService) in SingleDocumentBuilder.cs:67
  at void Build(IList<DocumentBuildParameters> parameters, string outputDirectory) in DocumentBuilder.cs:122
  at void BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, string baseDirectory, string outputDirectory, string templateDirectory) in DocumentBuilderWrapper.cs:42
  at string Exec(BuildJsonConfig config, BuildOptions options, string configDirectory, string outputDirectory) in RunBuild.cs:39
  at void <Execute>b__0() in DefaultCommand.cs:51
  at int Run(LogOptions options, Action run) in CommandHelper.cs:48
  at int Execute(CommandContext context, Options options) in DefaultCommand.cs:31
  at Task<int> Execute(CommandContext context, CommandSettings settings) in CommandOfT.cs:40
  at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext context, ITypeResolver resolver, IConfiguration configuration) in CommandExecutor.cs:166
@filzrev
Copy link
Contributor

filzrev commented Nov 17, 2024

Reported error seems to be reported by Jint JavaScript interpreter.

By default MaxParseDepth is configured as 64.
If this value is not sufficient for specific use case.
It need to increase this option value.

Is it able to provide max depth information?

@MiTschMR
Copy link
Author

From what I can see in the raw model json the maximum depth is 21 (42 spaces) when it fails in one occasion, another time it is 15 (30 spaces). The parsing stops in the middle of the line within the spaces, it doesn't even make it to the properties, like this:

                              *fails here on this space character*        "PvHarm": {
                                        "format": "double",
                                        "type": "number"
                                      },

@filzrev
Copy link
Contributor

filzrev commented Nov 18, 2024

I've created minimum projects that can reproduce problems.

Is it able to test required maxDepth with your swagger.json file?
And if possible, provide minimum swagger code fragment that can reproduce the problem.

Program.cs

using Jint;

const int maxDepth = 64;
var data = File.ReadAllText(@"C:\Temp\swagger.json");

var engine = new Engine(options =>
{
    options.Json.MaxParseDepth = maxDepth;
})
.SetValue("data", data);

var text = "return JSON.stringify(JSON.parse(data))";

var result = engine.Evaluate(text).AsString();
Console.WriteLine("Parse operation is successfully completed.");

Issue_10393.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Jint" Version="4.1.0" />
  </ItemGroup>

</Project>

@MiTschMR
Copy link
Author

I will try to do so when I am back, currently out of office. Thanks for the repro details!

@MiTschMR
Copy link
Author

Alright, the issue does not occur with the provided Program.cs. However, using the same swagger.json causes the exception in docfx:

[...]
Creating output...
Searching built-in plugins in directory C:\Program Files\docfx\...
7 plug-in(s) loaded.
Building 5 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>ValidateConceptualDocumentMetadata)...
Building 1 file(s) in RestApiDocumentProcessor(BuildRestApiDocument=>ValidateRestApiDocumentMetadata=>ApplyOverwriteDocumentForRestApi)...
Building 2 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
Building 5 file(s) in TocDocumentProcessor(BuildTocDocument)...
Building 1495 file(s) in ManagedReferenceDocumentProcessor(BuildManagedReferenceDocument=>SplitClassPageToMemberLevel=>ValidateManagedReferenceDocumentMetadata=>ApplyOverwriteDocumentForMref=>FillReferenceInformation)...
Applying templates to 1508 model(s)...
<path to swagger.json>: error ApplyTemplatePreprocessorError: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of JSON reached at position 1166075
   at parse RestApi.common.js:297:22
   at addDefinition (definition, definitions) RestApi.common.js:297:22
   at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
   at forEach RestApi.common.js:88:72
   at forEach RestApi.common.js:88:17
   at forEach RestApi.common.js:87:13
   at RestApi.common.js:86:9
DocumentException: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of JSON
reached at position 1166075
   at parse RestApi.common.js:297:22
   at addDefinition (definition, definitions) RestApi.common.js:297:22
   at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
   at forEach RestApi.common.js:88:72
   at forEach RestApi.common.js:88:17
   at forEach RestApi.common.js:87:13
   at RestApi.common.js:86:9
     DocumentException: Error transforming model "<path to raw model of swagger.json>" generated from "<path to swagger.json>" using "RestApi.html.primary.js". Error running Transform function inside template preprocessor: Error: Max. depth level of JSON
     reached at position 1166075
        at parse RestApi.common.js:297:22
        at addDefinition (definition, definitions) RestApi.common.js:297:22
        at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
        at forEach RestApi.common.js:88:72
        at forEach RestApi.common.js:88:17
        at forEach RestApi.common.js:87:13
        at RestApi.common.js:86:9
          InvalidPreprocessorException: Error running Transform function inside template preprocessor: Error: Max. depth level of JSON reached at position 1166075
             at parse RestApi.common.js:297:22
             at addDefinition (definition, definitions) RestApi.common.js:297:22
             at addComplexTypeMetadata (child, definitions) RestApi.common.js:339:9
             at forEach RestApi.common.js:88:72
             at forEach RestApi.common.js:88:17
             at forEach RestApi.common.js:87:13
             at RestApi.common.js:86:9
            at object TransformModel(object model) in PreprocessorWithResourcePool.cs:72
            at object TransformModel(object model) in Template.cs:93
            at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:97
       at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:114
       at void <ProcessCore>b__0(InternalManifestItem item) in TemplateProcessor.cs:190
       at void <RunAll>b__0(TElement s) in DocumentExceptionExtensions.cs:95
  at void RunAll<TElement>(IEnumerable<TElement> elements, Action<TElement> action, int parallelism, CancellationToken cancellationToken) in DocumentExceptionExtensions.cs:105
  at void RunAll<TElement>(IReadOnlyList<TElement> elements, Action<TElement> action, int parallelism, CancellationToken cancellationToken) in DocumentExceptionExtensions.cs:68
  at List<ManifestItem> ProcessCore(List<InternalManifestItem> items, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:185
  at List<ManifestItem> Process(List<InternalManifestItem> manifest, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:82
  at List<ManifestItem> ProcessTemplate() in ManifestProcessor.cs:186
  at void Process() in ManifestProcessor.cs:53
  at void Handle(List<HostService> hostServices, int maxParallelism) in LinkPhaseHandler.cs:32
  at Manifest Build(DocumentBuildParameters parameters, IMarkdownService markdownService, CancellationToken cancellationToken) in SingleDocumentBuilder.cs:71
  at void Build(IList<DocumentBuildParameters> parameters, string outputDirectory, CancellationToken cancellationToken) in DocumentBuilder.cs:124
  at void BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, string baseDirectory, string outputDirectory, string templateDirectory, CancellationToken cancellationToken) in DocumentBuilderWrapper.cs:42
  at string Exec(BuildJsonConfig config, BuildOptions options, string configDirectory, string outputDirectory, CancellationToken cancellationToken) in RunBuild.cs:39
  at void <Execute>b__0() in DefaultCommand.cs:51
  at int Run(LogOptions options, Action run) in CommandHelper.cs:48
  at int Execute(CommandContext context, Options options) in DefaultCommand.cs:31
  at Task<int> Execute(CommandContext context, CommandSettings settings) in CommandOfT.cs:40
  at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext context, ITypeResolver resolver, IConfiguration configuration) in CommandExecutor.cs:166

It again happens randomly in the line, not at a certain property. On another occasion it was in the middle of a property name. To me it doesn't make any sense why it would fail on one occasion and not on the other. I am not able to provide a minimal reproduction swagger.json until at earliest in a week, again out of office, though it will most likely not be easy to make one with just dummy classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants