BloopInstall is unreasonably slow for large builds

[SheliakLyr]

Describe the bug

For large gradle builds (hundreds of modules), bloop install takes a lot of time (at least more than 10 minutes for my case, I gave up after that), which makes it unusable when compared to the alternatives (IDEA import takes ~1 minute).

Quick look at the profiler is enough to locate to core issue. For each project, all other projects are fetched in order to match outputs to inputs between projects. The following code is executed for all projects, which does not scale:

  private def getArchiveSourceSetMap(
      rootProject: Project,
      sourceSets: Set[SourceSet]
  ): Map[File, SourceSet] = {
    val archiveSourceSets = for {
      project <- rootProject.getAllprojects.asScala
      archiveTask <- tasksWithType(project, classOf[AbstractArchiveTask])
      sourcePathObj <- getSourcePaths(archiveTask.getRootSpec())
      sourcePath <- sourceSets.find(_.getOutput == sourcePathObj)
    } yield archiveTask.getArchivePath -> sourcePath
    archiveSourceSets.toMap
  }

Expected behavior

BloopInstall should be reasonable fast for large repositories. At least as good as import to the Intellij IDEA.

Operating system

Linux

Version of Gradle

8.8

Version of gradle-bloop

2.13/1.6.2

Version of Bloop

unreleated

[Arthurm1]

Hi @SheliakLyr

10 mins seems far too long. I've run an import for 200 modules in 20ish seconds.

Intellij uses a different method than Bloop to extract the project data. Gradle runs theirs in parallel by default.

Could you try adding the following to your gradle.properties file and see if it makes a difference to the import time...

org.gradle.parallel=true

If it does not - could you create a sample repo of your build and I'll have a look at it.

[SheliakLyr]

Hi,

Unfortunatelly, I am already using parallel mode (however, with only 2 workers):

org.gradle.jvmargs=-Xmx4000m -Xss2m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -Dorg.gradle.java.compile-classpath-packaging=true
org.gradle.caching=true
org.gradle.parallel=true
org.gradle.priority=low
org.gradle.workers.max=2
org.gradle.cache.reserved.mb=2000
org.gradle.java.compile-classpath-packaging=true

About sample repo.. I can try to reproduce this issue on some generated, large projects. My build currently has 1041 modules. Each module has multiple source roots, which might make the issue worse.

I am also willing to personally contribute some changes and test them.

Hotspots from CPU profiling:

[SheliakLyr]

I created a simple gradle build: https://github.com/SheliakLyr/large_gradle_build_example
Number of projects can be customized by changing loop limiter in settings.gradle. By default, 600 projects are created.

bloopInstall takes a couple of minutes to execute. I could not bother to wait to have the exact time. My "real life" build is even larger (~1000 projects, bloop capable ~800).

I have also prepared experimental fixes that offer very big improvements.

First one: https://github.com/SheliakLyr/gradle-bloop/tree/performance makes the bloopInstall finish in ~3 minutes. Changes are very simple, basically I compute a list of projects only once per each task.

Second one: https://github.com/SheliakLyr/gradle-bloop/tree/performanceHacks speeds up the buildTask to 1m30s by sharing the list of projects between all tasks. I suspect this might be unsafe and/or broken.

[jvican]

The first change you made there looks interesting @SheliakLyr, could you submit it in a PR? Re the second, we'd need to take a closer look at whether that works.

I'm really surprised about your bloopInstall times though, I have a large build at $WORK and I see very fast executions of bloopInstall with Gradle 9 (you use 8.8 so it should be equivalent).