Tile Generation

Note: This example is among the first docs produced and reflected early usage. For most modern large-scale reconstructions, we export grayscale to neuroglancer precomputed or N5 and serve off those static files, usually in cloud storage. DVID can serve as a proxy to grayscale stored as neuroglancer precomputed volumes. Local storage is mostly used for the highly mutable data like segmentation, meshes, and annotations.

---

Once you have ingested a grayscale image volume (uint8blk type), you can create an instance of the imagetile type and generate multi-scale tiles. We could call this quadtree generation but you can specify different resolutions at each scale so it may not be exactly a reduction by 2 along each dimension as you increase scale.

Specify the multi-scale resolutions

First, you need to create a JSON file that specifies the resolution of tiles at each scale level. Here's an example for tiling a grayscale volume:

{
    "0": {  "Resolution": [10.0, 10.0, 10.0], "TileSize": [512, 512, 512] },
    "1": {  "Resolution": [20.0, 20.0, 20.0], "TileSize": [512, 512, 512] },
    "2": {  "Resolution": [40.0, 40.0, 40.0], "TileSize": [512, 512, 512] },
    "3": {  "Resolution": [80.0, 80.0, 80.0], "TileSize": [512, 512, 512] }
}

Each pixel in a level 0 tile is 10 (nm) across and should match the resolution of the underlying grayscale volume. Level 0 is the highest resolution view of the data. At level 1, we downsample by 2 so each pixel is now 20 nm. A single tile at this level would cover four tiles at level 0. From level 1 to level 2 we downsample by 2 so each pixel is 40 nm and one tile at this scale would cover 16 tiles at level 0.

You want to create a tile specification that makes sure at level 0 you have full resolution, so level 0 resolution should match the innate resolution of our ingested grayscale, and at the highest level we generate a tile that shows the entire breadth of our data. With four levels of scaling (levels 0-3), a single level 3 tile (given downsampling by 2 at each increasing level) will show 64 full-resolution tiles underneath. In the example specification above, each tile is 512x512 so the level 3 field of view will be 8x512 or up to 4096 pixels in each dimension.

Note that each specified Resolution and TileSize has three components because we could tile in XY, XZ, and YZ.

Generating the tiles (small dataset)

DVID's imagetile type has a built-in tile generation system. In the future, we'll provide some hooks to run a clustered tiler outside of DVID to free up DVID from excessive computation.

First, create an imagetile data instance, making sure to specify the source of its grayscale via the source option:

% dvid repo 3f8c new imagetile mytiles source=mygrayscale format=jpg

Then, start the tile generation by loading the tile specifications:

% dvid -stdin node 3f8c mytiles generate planes=xy < /path/to/tilespec.json

Generating the tiles (large dataset)

Depending on the size of the source data, the above tile generation can take quite a bit of time for a single DVID server. For example, multi-terabyte grayscale volumes can take days to generate just the XY tiles. Clearly, a clustered solution is needed to bring this down to hours. We typically use clustered tile generation at Janelia, storing tiles from each cluster server and then setting the metadata with a final POST to the imagetile API "metadata" endpoint. If using a distributed storage backend with its own API (e.g., Google Cloud Storage), you can parallelize tile generation by directly storing tiles into the storage backend after getting the necessary key via an imagetile "tilekey" endpoint GET.