Tiling, row major vs column major

[asultan123]

Descriptors are generally generated from the tiling of the weight matrix (at least for the 1x1 case). The assumption is that the tiling follows a row major-esqe execution order. I.e you finish a row of tiles first, going column by column, before transitioning to the next row. That's not the only way to do it though. Tiling iterations can follow a column-major order. I,e the same ifmap pixel could be re-read as new filters are loaded while keeping the same channel indicies. One could alternate between one and the other depending on the access cost of the ifmaps vs weights memories.

[asultan123]

Tiling order is tied to onboard psum storage requirements. If you're going to write back psum->ofmap data because you can't keep it on-chip then you need to be mindful of the number of filters in flight you have while computing the layer. The number of filters in flight depends on the filter parallelism you have, the number of filters in the layer, and the tile scheduler. A row-major tile schedule (i,e F, C loop order) will retire filters as fast as possible so they can be written back to memory. This will reduce the on-chip storage requirement significantly. A column-major order (i.e C, F loop order) will retire filters as late as possible thus maximizing required on-chip storage. ALAP tile scheduling is fine if you have enough on-chip storage. In fact, you get two nice bonuses when doing ALAP. You can move relevant ifmap channels being processed to a lower level in the memory hierarchy to access it more frequently as you process filters. The lower level memory would then be much smaller and the access cost would be much lower. Additionally, if you can store the full ofmap as well as the ifmap you can "flip" the memories and keep all intermediate results between layers on-chip. This can affect latency if you're DRAM is slow, or at the very least keep the energy access cost for if map down.