Offline tiles in a webapp

[pietervdvn]

Hi,

I'd like to add an "offline mode" to MapComplete, using PMTiles. I'd extract some data from the full PMtiles archive, depending on the area the user wants to have locally.

I'm now wondering what the best approach for this would be:

1. Download all tiles on zoomlevel 14 as needed and store them locally (+ which other zoomlevels should I need?)
2. Extracting a part as PMTiles and then somehow host them via JS

What would be the most appropriate route to take?

[wipfli]

I think a good approach that a heard from someone else for offline usage is the following:

1. Take your planet pmtiles file and extract all zoom 8 subpyramids into separate pmtiles file.
2. Store the subpyramids somewhere as separate files and let the user download them as needed.
3. In the client, you simply have to manage a few zoom 8 subpyramids.

The filenames could be something like 8-134-89.pmtiles i.e. z-x-y.pmtiles. This one for example would cover Zürich and surroundings:

https://maps.protomaps.com/#map=8.08/47.215/8.143&theme=light&lang=en&tiles=https://oliverwipfli.ch/v4.pmtiles&flavorName=light&show_boxes=true

You can estimate the size of those subpyramids like this:

With maxzoom 15 you have 2 ** (2 * (15 - 8)) = 16384 tile at maxzoom per subpyramid. If every tile is 100 kilobytes, that gives 1.6 gigabytes plus the lower zoom tiles, so maybe a bit more than 2 gigabytes. Maybe this is too large of a file for offline users to download, so you could choose a higher zoom level for your subpyramids like maybe zoom 10, then it would be 100 megabytes for the maxzoom...

Anyway, this is not a super smart approach, but the logic will be super easy to implement and maintain.

[pietervdvn]

Actually a pretty good idea!

[pietervdvn]

Is there a clean command to extract a specific sub-pyramid?

[wipfli]

I think you can do it with the python library.

First you need a way to iterate over the tiles of the subpyramid, maybe with something like this:

parent_tile = mercantile.Tile(x=134, y=89, z=8)
for child_z in range(parent_tile.z, 16):
  for child in mercantile.children(parent_tile, zoom=child_z):
    # extract from global pmtiles file
    # write to subpyramid

then you need to read a tile from the archive and write one again. The roundtrip test gives you an idea of how to do it:

PMTiles/python/pmtiles/test/test_reader_writer.py

Lines 9 to 32 in 667f0fd

	def test_roundtrip(self):
	buf = BytesIO()
	writer = Writer(buf)
	writer.write_tile(zxy_to_tileid(0, 0, 0), b"1")
	writer.write_tile(zxy_to_tileid(1, 0, 0), b"2")
	writer.write_tile(zxy_to_tileid(2, 0, 0), b"3")
	writer.finalize(
	{
	"tile_compression": Compression.UNKNOWN,
	"tile_type": TileType.UNKNOWN,
	},
	{"key": "value"},
	)
	reader = Reader(MemorySource(buf.getvalue()))
	self.assertEqual(reader.header()["version"], 3)
	self.assertEqual(reader.header()["min_zoom"], 0)
	self.assertEqual(reader.header()["max_zoom"], 2)
	self.assertEqual(reader.header()["clustered"], True)
	self.assertEqual(reader.metadata()["key"], "value")
	self.assertEqual(reader.get(0, 0, 0), b"1")
	self.assertEqual(reader.get(1, 0, 0), b"2")
	self.assertEqual(reader.get(2, 0, 0), b"3")
	self.assertEqual(reader.get(3, 0, 0), None)

so we would do:

tile_bytes = reader.get(child.z, child.x, child.y)
writer.write_tile(zxy_to_tileid(child.z, child.x, child.y), tile_bytes)

the writer should look something like

from pmtiles.tile import zxy_to_tileid, TileType, Compression
from pmtiles.writer import Writer

with open(out_filepath, 'wb') as f1:
    writer = Writer(f1)
    max_z = 0
    # do the loop here over the child tiles of the pyramid

    bounds = mercantile.bounds(parent_tile)
    min_lon_e7 = int(bounds.west * 1e7)
    min_lat_e7 = int(bounds.south * 1e7)
    max_lon_e7 = int(bounds.east * 1e7)
    max_lat_e7 = int(bounds.north * 1e7)

    writer.finalize(
        {
            'tile_type': TileType.WEBP, 
            'tile_compression': Compression.NONE,
            'min_zoom': parent_tile.z,
            'max_zoom': 15,
            'min_lon_e7': min_lon_e7,
            'min_lat_e7': min_lat_e7,
            'max_lon_e7': max_lon_e7,
            'max_lat_e7': max_lat_e7,
            'center_zoom': int(0.5 * (parent_tile.z + 15)),
            'center_lon_e7': int(0.5 * (min_lon_e7 + max_lon_e7)),
            'center_lat_e7': int(0.5 * (min_lat_e7 + max_lat_e7)),
        },
        {
            'attribution': 'your attribution'
        },
    )

[bdon]

Download all tiles on zoomlevel 14 as needed and store them locally (+ which other zoomlevels should I need?)

For the best interactive use cases you should store a full subpyramid up to the zoom 0 tile.

Extracting a part as PMTiles and then somehow host them via JS

How large do you intend to allow the end user to download? A neighborhood, a city, or country?

There is an efficient algorithm for extracting a subset of a PMTiles archive for any polygonal area that consumes little memory for hundreds of millions of tiles, by batching tiles that are adjacent on the space-filling curve. The only implementation I know if is the one in the go-pmtiles CLI: https://github.com/protomaps/go-pmtiles/blob/main/pmtiles/extract.go#L237

This is difficult to port to JavaScript because it depends on Roaring Bitmaps. If your offline area is only a few hundred tiles you may be able to get away with just downloading each of them individually.