Small benchmark section in the README

[laem]

Hi,

Would it be possible to list a few example runs of tilemaker to let the user know what's last tested compute time for example regions ?

E.g. for a country, for europe, for planet / for a high-ram machine and a low ram machine

In order to let users know what time to expect.

I've found interesting info in the issues but they can date back to 2021 and older versions of tilemaker, it's hard to know what's the current status.

I'll let you know mine in the coming days, since I'm coding a script to spawn Scaleway compute instances dedicated to building tiles.

[laem]

Here is a 300 Go RAM machine in the "Reading / Block" phase for europe.osm.pbf with the --fast option.

Only 30 Go of RAM are used, but all cores are mobilized to 100%.

Edit : after a few minutes, the memory used grows to 65 Go. From what I understand, Tilemaker reads the osm data by writing it to the RAM.

From what I understand, --fast does not activate --no-compress-nodes and --no-compress-ways. I'll try with those to see the difference.

Edit : 105 G of RAM used 🌶️

When this text pops,

Block 54702/54703 (971727 ms)
SortedWayStore: 21392 groups, 1921603 chunks, 18244648 ways, 574775724 nodes, 1500368604 bytes

144 Go of RAM is used.

The "Creating mbtiles at /root/europe.mbtiles" happens just after and the RAM usage looks to end its strong increase and just gain some Go slowly.

After ~20 minutes the europe.mbtiles is alreay 20 Go.

After about ~30 minutes I get the message "Filled the tileset with good things at /root/europe.mbtiles".

Well I only saw the "z6" step, I thought there would be other steps.

Now starting the final process of my ansible script : pmtiles/pmtiles convert europe.mbtiles europe.pmtiles.

The process is single-core and uses only < 2 Go of RAM. What a waste 😅 cc @bdon this benchmark could be interesting for you.

It took about 5 minutes.

Looks like a success ! Let's pretend we used the "POP2-HM-32C-256G" scaleway workload instance. This would have cost us 0,8 €. That's rather sobre.

[laem]

Let's go for the planet generation now.

Download as a torrent to avoid charge and go faster.

aria2c --seed-time=0 https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf.torrent

Took 5 minutes.

But the file is corrupt or incompatible with Tilemaker :/

Reading .pbf /root/planet-latest.osm.pbf
terminate called after throwing an instance of 'std::runtime_error'
what(): readBlob: unexpected eof
Aborted (core dumped)

Oh, my bad ! The torrent was "planet-250714.osm.pbf", the one tested above was the terminated wget test.

OK let's go !

Planet is a bit less that 3x Europe. We can expect 390 Go of RAM to be used, so the RAM should reach saturation (by almost nothing) at the end. We'll see and try with a stronger server in case.

We'd also need to try to solve this first "warning: PBF has very large blocks, which may slow processing
to fix: osmium cat -f pbf your-file.osm.pbf -o optimized.osm.pbf".

And output a .pmtiles directly, I just found out that it's possible. Wonder why I went with go-pmtiles then.

OK, the planet.mbtiles was written in about 1h, I didn't precisely time it. The gopmtiles convert command looks way slower than for europe, however. I should definitely try to avoid this step.

Trying a new run with .pmtiles and --no-compress-x got the process killed when it reached the full machine RAM !

Block 16777/16778 (2509302 ms)
SortedWayStore: 21495 groups, 2925642 chunks, 35068423 ways, 1398660043 nodes, 6361759558 bytes
Block 1/429 Killed

Tried osmium cat -f pbf your-file.osm.pbf -o optimized.osm.pbf.

It takes 13 minutes. The server's resources are not used a lot. 🤔 I don't believe it could help us given the process takes 30 minutes without this.

Trying osmium cat -f pbf planet.osm.pbf -o planet-optimized.osm.pbf at ⏲️ 16h52.

The process is slow too, taking only 5 % of RAM and some cores, and would take about ~ 20 minutes given the displayed progress percentages.

Starting without the optimisations and without de compress options at 17h04.

🏁 peak at 364 Go RAM, took 46 minutes before the "using dense index for pmtiles", then peaked at 368 Go , currently writing pmtiles using full server resources.

We'll then have to check if the pmtiles produced are the same as the gopmtiles convert ones.

In the end, a bit more than one hour of processing for planet for 3 € € seems to be the big conclusion given the current Tilemaker / Scaleway tech.

We'll then have to check if using a machine with less RAM makes us lose a linear time, or not. In this cas, it's not interesting, they cost only ~ linearly less.

[etienneJr]

And output a .pmtiles directly, I just found out that it's possible. Wonder why I went with go-pmtiles then.

The pmtiles generated by tilemaker is not clustered. Whereas it is with the 2 steps strategy (at least using pmtiles convert command).
It means that :

• it is probably larger (not clear to me to which extent)
• it makes issues for nginx ?
• the command pmtiles extract won't work

See #653
So I suggest you keep the 2 steps strategy 😉

Another 2 steps strategy could be using the recent pmtiles cluster command on the unclustured pmtiles made by tilemaker. No idea at all if it will be faster.

[laem]

Thanks !
89520043693 for 2 step.
90107794489 for 1 step tilemaker only.

[systemed]

Really interesting - thanks for this. I agree a section in the README would be a good idea.

[laem]

Probable consequence of this cluster matter (that I do not grasp), the pmtiles built by tilemaker version serves a 40 Mo tile on my map. The go-pmtiles version does not.

[systemed]

Hm, that sounds like a bug.

[laem]

Could be on my side, in the nginx serving ? You can reproduce it here. https://pmtiles.io/#url=https%3A%2F%2Fserveur.cartes.app%2Fgtfs%2Fplanet-tilemaker.pmtiles&map=11.6/47.3196/-3.1499

[laem]

I'm trying the second option : Tilemaker to .pmtiles, and then "go-pmtiles cluster" command.

Do you think this option can be set to true with the output of Tilemaker ?

--no-deduplication: Do not attempt to de-duplicate tile contents. Use this to speed up cluster if you know the input has only unique tiles.

It appears for now that the cluster operation on planet will take about 20 minutes. Which is OK, but if it can be accelerated, that's even better !

[systemed]

Yes, that option should be fine!

[bdon]

40 Mo tile on my map. The go-pmtiles version does not.

Clustered or unclustered should not affect the contents or size of any individual tile. You could see the unclustered version in the archive structure inspector at https://pmtiles.io/archive/ and see if a leaf directory is unusually large: leaves have runlength 0 (leaf) and length is the size in bytes.

[laem]

OK

This is the planet.pmtiles resulting from Tilemaker to MBTiles then pmtiles convert : https://pmtiles.io/#url=https%3A%2F%2Fserveur.cartes.app%2Fgtfs%2Fplanet.pmtiles

This is the result of Tilemaker to PMTiles + pmtiles cluster :
https://pmtiles.io/#url=https%3A%2F%2Fserveur.cartes.app%2Fgtfs%2Fplanet-test.pmtiles

There's a week of data between both. Everything looks good, only a couple of kb of difference between both.

[etienneJr]

in the archive structure inspector

@bdon I can only see 3501 entries, is it normal? In any case, even if the 88 million tiles were displayed, it would be impossible to find the few problematic tiles in such a long list, wouldn't it?

This is the result of Tilemaker to PMTiles + pmtiles cluster

Clustered or unclustered should not affect the contents or size of any individual tile.

@laem Did you check the (unclustered) pmtiles made by tilemaker before clustering? did it still include or serve 40 Mo tiles? It should not include 40 Mo tiles since the clustered version is clean, so is it a problem while serving the tile?

only a couple of kb of difference between both.

what about calculation time? Is one strategy faster than the other?

[bdon]

I can only see 3501 entries, is it normal?

Yes, because those are only the root entries pointing to leaves, you can click into each row to see the actual tile entries.

even if the 88 million tiles were displayed, it would be impossible to find the few problematic tiles in such a long list, wouldn't it?

You can use the python pmtiles package to parse the directories and find anything usual.

[laem]

@laem Did you check the (unclustered) pmtiles made by tilemaker before clustering? did it still include or serve 40 Mo tiles? It should not include 40 Mo tiles since the clustered version is clean, so is it a problem while serving the tile?

No. I did not try to use these "-test" tiles yet on dev.cartes.app. Just did, up in 5 minutes !

what about calculation time? Is one strategy faster than the other?

Yes, I found pmtiles convert very slow on the whole planet, too slow. Maybe even slower than the Tilemaker to mbtiles step ! I don't have the exact figures though.

Edit : looks good !

First capture : session starting at the France zoom.

Second capture : continuing the session and zooming to the max tile zoom. "only" 5 Mb used 👍