neureality --model --image
models supported --faster-rcnn --mask-rcnn
Some of the data can be seen in tensorboard, by running: tensorboard --logdir /neuReality/runs. Such as image bounding boxes and image mask ( for mask rcnn).
I also shows an image of profiler output (there is no nice integration of profiler to tensorboard as in tensorflow).
As image is quite large you may need to click on it to see it properly.
The profiler output in tensorfboard has only profiling for layers and preprocess/inference/postprocess, the inference is devided to
backbobe, rpn, roi-heads.
But he preffered way to see the profiling resulys is by viewing the trace file by opening chrome://tracing in Chrome browser and loading the trace.json file that was generated by the profiler.
The code wraps all forward functions of the modules with profiler, so in trace you have all granularities.
For more info https://www.chromium.org/developers/how-tos/trace-event-profiling-tool
I used pytorch profiler (https://pytorch.org/tutorials/recipes/recipes/profiler.html). The two methods that i came up for profiling are:
-
Patching the code that needs to be profiled and adding profiler with statement to pieces that need to be profiled. This enabled better
profiling granularity, as you can add statements to whatever you want, but you need to write code for it and patch library functions.
-
The second method is wrapping forward function of each layer (module without children) with profier. You can also choose wo trap
all modules. This is done aotomalicaly and you don't have to change a code for it and don't have to support it.
It is better to run the model on few hundred images and look at the average cpu running time (CPU time avg in profiler).
Torch profiler has profile memmory functionality, but to use it i had to upgrade to newer version
torch 1.4.0 torchvision 0.5.0 tensorboard 2.0.0