Parallel drc implementation

[Scafir]

/!\ must be merged at the same time as fossi-foundation/globalfoundries-pdk-libs-gf180mcu_fd_pr#15

This is an upstream MR based on: wafer-space/gf180mcu#26

Ongoing discussion with klayoutmatthias can be found here

Here is the original message:

Here is a mockup of what a nicely integrated, parallel DRC check in klayout could look like.
While it works flawlessly, there is room for a bit of discussion (see below). One implementation point that stands out is that klayout is not very flexible in the way it deals with drc reports, which causes a bit of gymnastics that should ideally be integrated in klayout core rather than here.

It works well, with significant performance improvement similar or better than run_drc.py and much better than just the previous gf180mcu.drc. This is possible because one-rule-after-another cannot maintain CPUs fed throughout the execution. Worse yet, it seems that when run from the gui, only a single core at most was be used (as opposed to starting it from command line). The performance is similar to the python approach, but with the significant advantage of generating a single report file. It should actually be a little faster, as the general layer calculation are only done once with the present method.

Some light gymnastics / hacks are required to end up with a single report and display it through the gui.

This implementation suffers from a few disadvantages:

• As in run_drc.py, unnecessary files are created to hold partial drc checks. It is not possible to change this without modifying klayout core.
• For reasons that I ignore, forking in the gui thread seem to have issues that the command line approach does not have. For gui, I thus spawn a new instance of klayout from command line. This has little impact on large designs, but small design drc execution time will be lower bounded by it
• The klayout gui interface simply hangs while the check is ongoing. Before the check would at least provide some amount of GUI updates.
• WaiveDB is not implemented (can be added)
• A tmp directory is created to hold temporary files. It is currently not cleared at the end of execution. (can be added)

Note that run_drc.py is not affected, as it uses the files in macros/run_drc_main

I will publish some benchmark numbers tomorrow. So far I've seen 4x improvements in speed for a gui check for a middle size design.

Benchmark

Run time:

Design	New gf180mcu.drc (32 workers) [s]	Old gf180mcu.drc [s]	run_drc.py (32 threads, patched for dummy tests) [s]
Trivial Design (bondpad)	2.6	2.94	7.06
Simple full chip design (unfilled)	46.51	285.72	43.09
Simple full chip design (filled)	248.91	1351.2	256.4

Commands

For new gf180mcu.drc: `time klayout -b -r /path/to/gf180mcu.drc -rd input=/path/to/layout.gds -rd feol=true -rd beol=true -rd conn_drc=true -rd dummy=true -rd offgrid=true -rd run_mode=deep -rd out=a.lyrdb` For old gf180mcu.drc: `time klayout -b -r /path/to/gf180mcu.drc -rd input=/path/to/layout.gds -rd feol=true -rd beol=true -rd conn_drc=true -rd dummy=true -rd offgrid=true -rd run_mode=deep` For run_drc.py: `time python3.13 /home/claforge/Documents/gf180mcu/gf180mcuD/libs.tech/klayout/tech/drc/run_drc.py --path=/home/claforge/cernbox/CERN/PhD/gf180mcu-project-trident-gf180-teststructure/klayout/top.layout.gds.gz --variant=D --mp 32 --run_mode=deep`

Ram usage:

Design	New gf180mcu.drc (32 workers) [Gb]	Old gf180mcu.drc [Gb]	run_drc.py (32 threads, patched for dummy tests) [Gb]
Simple full chip design (unfilled)	5.6	2.3	12

Ran on an AMD Ryzen 9 9950X CPU (32 cores)

The main difference in terms of performance between my implementation and run_drc.py is that my implementation runs the preliminary calculations once, and then forks. The run_drc.py runs everything in parallel. Depending on the number of tests to run, the cost of the preliminary calculation and the number of threads this can be good or bad in terms of runtime.

Blockers:

• Implement workers argument for number of threads to use
• Check if file is saved
• Use report for the output
• Add vanilla flow in case of a single worker
• Add gf180mcu.drc documentation for options
• Add parallel_rule_checker.rb documentation?
• Test the merging of reports properly
• Add cancel functionality

[Scafir]

A discussion about the missing klayout features can be found in https://klayout.de/forum/discussion/2839/drc-scripts-reports-and-parallelism
It seems that on-paper, the present approach is fine (as long as we run the test in a batch mode klayout process).

We just had a discussion with @mole99, and we've decided to push this forward. What is currently missing is:

• Tests for the merging of reports
• Use dummy drc check for supporting cancel functionality
• Ensure that sub-processes are properly killed on ctrl+c in command and when cancelled in gui

As @mole99 wants to implement a more elegant way to have specify what tests should be run, which would be built on some of the present modifications, I'll make a merge request in the coming days for adding the structural changes, without enabling the parallel processes as a first step.