Help: Oops! A fatal internal error occurred (ArrayIndexOutOfBoundsException) #16888

mindcrunch4u · 2024-07-01T23:55:03Z

mindcrunch4u
Jul 1, 2024

Hello!

I am running the cpp-security-extended.qls on a 20GB database

CodeQL-CLI: version 2.17.4
command:
- codeql database analyze codeql_db path/to/1.0.0/codeql-suites/cpp-security-extended.qls --format=csv --output =20240629_result.csv --threads 32 --ram 102400
- codeql database analyze codeql_db path/to/1.0.0/codeql-suites/cpp-security-extended.qls --format=csv --output =20240629_result.csv

Both commands (with or without threads and ram options) had the same result.

The output (without --threads option, and --ram set to 102400):

(omitted)
[83/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Best Practices/Likely Errors/CommaBeforeMisleadingIndentation.ql.
[84/91] Loaded /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/MissingCheckScanf.qlx.
[85/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/SizeCheck.ql.
[86/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/SizeCheck2.ql.
[87/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/FailedExtractorInvocations.ql.
[88/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/ExtractionWarnings.ql.
[89/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/ExtractedFiles.ql.
[90/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Summary/LinesOfCode.ql.
[91/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Summary/LinesOfUserCode.ql.
Starting evaluation of codeql/cpp-queries/Critical/DoubleFree.ql.
Oops! A fatal internal error occurred. Details:
com.semmle.util.exception.CatastrophicError: An error occurred while evaluating DataFlowImplCommon::Cached::edgeOrder/1#6d956c05/2@43a39avp
java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 1506635235 out of bounds for long[224938403]
The RA to evaluate was:

    {2} r1 = AGGREGATE `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_range`, `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_term` ON In.5, In.6, In.7 WITH RANK<1 ASC2 ASC> OUTPUT , Agg.0, Agg.1
    return r1

(eventual cause: ArrayIndexOutOfBoundsException "arraycopy: last destination index 1506635235 out of bounds for long[224938403]")
        at com.semmle.inmemory.pipeline.PipelineInstance.wrapWithRaDump(PipelineInstance.java:168)
        at com.semmle.inmemory.pipeline.PipelineInstance.exceptionCaught(PipelineInstance.java:152)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:593)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:410)
        at com.semmle.inmemory.scheduler.IntensionalLayer$IntensionalWork.evaluate(IntensionalLayer.java:93)
        at com.semmle.inmemory.scheduler.SimpleLayerTask$SimpleLayerWork.doWork(SimpleLayerTask.java:69)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
        at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:677)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 1506635235 out of bounds for long[224938403]
        at java.base/java.lang.System.arraycopy(Native Method)
        at com.semmle.inmemory.eval.aggregate.TupleListList.addList(TupleListList.java:44)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.commitCurrentRun(AggregateEvaluator.java:463)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator$GroupAndJoin.addTuple(AggregateEvaluator.java:512)
        at com.semmle.inmemory.eval.CancelCheckingSink.addTuple(CancelCheckingSink.java:18)
        at com.semmle.inmemory.relations.BaseIntArrayRelation.map(BaseIntArrayRelation.java:84)
        at com.semmle.inmemory.caching.PagedRelation.map(PagedRelation.java:158)
        at com.semmle.inmemory.relations.AbstractRelation.deduplicateMap(AbstractRelation.java:138)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.evaluate(AggregateEvaluator.java:256)
        at com.semmle.inmemory.pipeline.AggregateStep.generateTuples(AggregateStep.java:43)
        at com.semmle.inmemory.pipeline.SimpleHeadStep.lambda$forwardInitialize$0(SimpleHeadStep.java:32)
        at com.semmle.inmemory.pipeline.HeadEndDispatcher.headEndWork(HeadEndDispatcher.java:75)
        at com.semmle.inmemory.pipeline.PipelineState.doSomeWork(PipelineState.java:88)
        at com.semmle.inmemory.pipeline.PipelineInstance.doWork(PipelineInstance.java:117)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
        ... 7 more

Seems like many queries have been completed, but this one triggers an exception.

Any suggestion is appreciated!

Answered by aschackmull

Jul 16, 2024

We've implemented most of what's needed to avoid the totalorder predicate in future versions, and the rest of the work will happen as soon as the relevant people are back from summer vacation.

This work is now complete, and should be part of 2.18.1 once that releases (likely around July 26th).

View full answer

smowton · 2024-07-02T13:34:09Z

smowton
Jul 2, 2024
Maintainer

Could you check if this is fixed with 2.17.5? There's a possibility the snag has already been fixed.

3 replies

mindcrunch4u Jul 3, 2024
Author

Hello @smowton !

Tried with 2.17.6 (since it's the latest version)

Now I get this error:

Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-497/PotentiallyExposedSystemData.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-416/IteratorToExpiredContainer.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-732/UnsafeDaclSecurityDescriptor.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-497/ExposedSystemData.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-807/TaintedCondition.ql.
Oops! A fatal internal error occurred. Details:
com.semmle.util.exception.CatastrophicError: An error occurred while evaluating DataFlowImplCommon::Cached::edgeOrder/1#6d956c05/2@e09049d4
java.lang.IllegalArgumentException: Maximum array length exceeded: 2147483640
The RA to evaluate was:

    {2} r1 = AGGREGATE `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_range`, `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_term` ON In.5, In.6, In.7 WITH RANK<1 ASC2 ASC> OUTPUT , Agg.0, Agg.1
    return r1

(eventual cause: IllegalArgumentException "Maximum array length exceeded: 2147483640")
        at com.semmle.inmemory.pipeline.PipelineInstance.wrapWithRaDump(PipelineInstance.java:168)
        at com.semmle.inmemory.pipeline.PipelineInstance.exceptionCaught(PipelineInstance.java:152)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:593)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:410)
        at com.semmle.inmemory.scheduler.IntensionalLayer$IntensionalWork.evaluate(IntensionalLayer.java:93)
        at com.semmle.inmemory.scheduler.SimpleLayerTask$SimpleLayerWork.doWork(SimpleLayerTask.java:69)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
        at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:677)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: Maximum array length exceeded: 2147483640
        at com.semmle.inmemory.eval.aggregate.TupleListList.checkSize(TupleListList.java:41)
        at com.semmle.inmemory.eval.aggregate.TupleListList.prepareForAdd(TupleListList.java:32)
        at com.semmle.inmemory.eval.aggregate.TupleListList.addList(TupleListList.java:59)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.commitCurrentRun(AggregateEvaluator.java:463)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator$GroupAndJoin.addTuple(AggregateEvaluator.java:512)
        at com.semmle.inmemory.eval.CancelCheckingSink.addTuple(CancelCheckingSink.java:18)
        at com.semmle.inmemory.relations.BaseIntArrayRelation.map(BaseIntArrayRelation.java:84)
        at com.semmle.inmemory.caching.PagedRelation.map(PagedRelation.java:158)
        at com.semmle.inmemory.relations.AbstractRelation.deduplicateMap(AbstractRelation.java:138)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.evaluate(AggregateEvaluator.java:256)
        at com.semmle.inmemory.pipeline.AggregateStep.generateTuples(AggregateStep.java:43)
        at com.semmle.inmemory.pipeline.SimpleHeadStep.lambda$forwardInitialize$0(SimpleHeadStep.java:32)
        at com.semmle.inmemory.pipeline.HeadEndDispatcher.headEndWork(HeadEndDispatcher.java:75)
        at com.semmle.inmemory.pipeline.PipelineState.doSomeWork(PipelineState.java:88)
        at com.semmle.inmemory.pipeline.PipelineInstance.doWork(PipelineInstance.java:117)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
        ... 7 more

Any idea how should I proceed?

smowton Jul 4, 2024
Maintainer

I'm afraid at least for now you have simply exceeded our maximum problem size, specifically by making us try to compute a ranking larger than the maximum Java array size. For the time being I'm afraid the only solution would be to bound the scope of analysis more tightly, typically by considering more components of your software opaque external libraries (things that are only seen as binary objects or libraries during database creation), and less of them the source code that is subject to analysis (compiled from source during database creation).

As an example, codeql database create -l cpp -c "gcc test.c test2.c -o binary" might be reduced in scope like ... make test2.c into a library test2.a ...; codeql database create -l cpp -c "gcc -c test.c -ltest2 -o binary" -- so that test.c is analysed in full depth, while the functions in test2.c are considered opaque. The downside is that data-flow queries will only find results that do not originate, flow through or terminate inside test2.c, so this consideration should inform what subset of your source you consider important to build from source and therefore analyse in depth.

mindcrunch4u Jul 6, 2024
Author

@smowton thank you for the advice!

jbj · 2024-07-04T14:36:58Z

jbj
Jul 4, 2024
Maintainer

I'm afraid you've run into scalability problems with the totalorder predicate that was introduced in 2.17.4. If you downgrade to 2.17.3 it should work. We've implemented most of what's needed to avoid the totalorder predicate in future versions, and the rest of the work will happen as soon as the relevant people are back from summer vacation.

3 replies

mindcrunch4u Jul 6, 2024
Author

@jbj thank you! will give 2.17.3 a try, and I wish the relevant people a nice summer vacation!

aschackmull Jul 16, 2024
Maintainer

We've implemented most of what's needed to avoid the totalorder predicate in future versions, and the rest of the work will happen as soon as the relevant people are back from summer vacation.

This work is now complete, and should be part of 2.18.1 once that releases (likely around July 26th).

Answer selected by mindcrunch4u

mindcrunch4u Jul 18, 2024
Author

@aschackmull thank you! will give 2.18.1 a try once it's out!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Help: Oops! A fatal internal error occurred (ArrayIndexOutOfBoundsException) #16888

{{title}}

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Help: Oops! A fatal internal error occurred (ArrayIndexOutOfBoundsException) #16888

mindcrunch4u Jul 1, 2024

Replies: 2 comments · 6 replies

smowton Jul 2, 2024 Maintainer

mindcrunch4u Jul 3, 2024 Author

smowton Jul 4, 2024 Maintainer

mindcrunch4u Jul 6, 2024 Author

jbj Jul 4, 2024 Maintainer

mindcrunch4u Jul 6, 2024 Author

aschackmull Jul 16, 2024 Maintainer

mindcrunch4u Jul 18, 2024 Author

mindcrunch4u
Jul 1, 2024

Replies: 2 comments 6 replies

smowton
Jul 2, 2024
Maintainer

mindcrunch4u Jul 3, 2024
Author

smowton Jul 4, 2024
Maintainer

mindcrunch4u Jul 6, 2024
Author

jbj
Jul 4, 2024
Maintainer

mindcrunch4u Jul 6, 2024
Author

aschackmull Jul 16, 2024
Maintainer

mindcrunch4u Jul 18, 2024
Author