Upgrade mishaps

Last weekends Ubuntu upgrade to 18.04 LTS went mostly well, the Mapnik Python Binding problem from 17.10 has been fully fixed etc.

What didn’t go so well though was the database upgrade from PostgreSQL 9.6 / PostGIS 2.3 to 10 / 2.4.

irst of First of all the process for upgrading PostgreSQL and PostGIS to a new version at the same time is not as trivial as it could be, and requires some file name faking hacks before running the actual pg_upgrade:


Once I found that information the actual pg_upgrade ran fine, and so did the extension upgrade for PostGIS, too.

Map rendering also looked fine, but then when I tried to reactivate import of minutely diff jobs that should only take secondes, or minutes at best, just stalled, and eventually made one postgresql process run out of memory after allocating over 60GB of RAM over some 20 minutes.


In the end we came to the conclusion that something must already have gone wrong on the previous full planet import, somehow not causing problems on the older versions but triggering somethng strange on the more recent ones.

I dind not put much more effort into debugging this at that point, and to prevent possible spread of bad data decided to cut my losses and start a new full import right away without pre-announcing a downtime notice a week ahead of time.

Fortunately that import went well, and so does import of minutely diffs now. The system has since caught up and is less than 15 minutes behind on averaege.

I also have a gut feeling that rendering jobs are processed a little faster now, but unfortunately didn’t properly preserve timing results from test runs yet.

Rendering maps again

The database import has finished and the rendering daemon is running again.

The Openstreetmap and German Carto stiles have been updated to their most recent releases, so rendererd maps should finally look the same as on openstreetmap.org and .de again. The rest of the styles will follow today.

Database updates still need some work, so for now map data is of last Thursday, the time of the last full planet export.

Upcoming new database import

I will have to do another full import again, causing about a day of outage and some more before the MapOSMatic service instance is fully up to data again.

Starting tonight (Friday April 13th, 2018) at 21:00 UTC the GIS database will be offline, so no new rendering jobs will be processed until for a while. The web frontend will still work most of the time, already rendered Maps will still be available, and submission of new jobs may be possible, but these will only be queued then, and will only be processed after the database import, and some additional changes like stylesheet updates, have finished.

The reason this time is not that I did something wrong on the previous import, or on the minutely diffs import to keep the database up to date. This time it is neAeded to apply the necessary schema changes needed to finally support current OSM Carto default style releases again. OSM Carto v4.0 changed the database requirements in a non-backwards-compatible way, so a re-import is needed to not be stuck on v3.x forever, and to make the default style look the same as on openstreetmap.org again.

To be able to still support all the other older styles again, and to provide better internatiolization and localization features in the near future, I’ll actually be using the hstore-only schema used by the current German OSM style maintained by Sven Geggus, with database views on top that then provide the right schema for each style.

All this works fine in my test environment already, it just needs to be applied to the big machine now, and the re-import is the biggest step of that. I’ll be working on this during the upcoming Berlin OSM Hackweekend, and hope to have everything up and running again Saturday evening, but it may take parts of Sunday, too.

Natural sorting

So far street names that start or end in a number were sorted in simple textual order, e.g. 1-10-11-…-19-2-20-21-…

Now, with the help of the Python “natsort” package, these will be listed in “natural” order, like 1-2-…-10-11-…-19-20-21-…

At first this only worked for ASCII / Latin numbers, but with two little fixes to the “natsort” package it should now work for all numeric unicode characters (tested with Persian/Farsi so far).