March 31, 2013
The problem with OpenStreetMap multipolygons

Here’s a very simple example of why I think OpenStreetMap multipolygons are a disaster.

Let’s say you download a data extract of an area and you get a relation that has six member ways (all of them in outer role). The way most extracts work, you only get those ways that cross (or are inside) the download bounding box, in our case there are only two such ways:

situation

What now? In normal situations you would be able to clip the two ways using the bounding box and then connect them along the box. But the trouble is, you have two different (and equally valid) possibilities:

solution 1

solution 2

How do you decide which is the correct solution, is the body of the polygon in the center between the two ways or is it on opposite sides? Let’s see what options we have:

  • Orientation of ways: no help here - ways (and rings) can be oriented any way you like in OSM multipolygons. Usually in GIS and graphics polygons follow a clockwise/anti-clockwise rule, but in OSM this practice was abandoned. The only reason for this I could think of is to make it easier for the mappers to edit maps.
  • Member roles: in this case the outer/inner roles don’t help very much.
  • Tagging: I wouldn’t want to get into that mess. And there’s no general tagging solution for this problem, as far as I know.
  • Requiring that extracts contain all the relation’s members: OSM has a lot of very large relations that span hundreds of kilometers and even whole continents. Why would I want to have all of the Russia’s coastline when I download the extract of, say, small part of Estonia?

Don’t be fooled by the simplicity of my example into thinking that this is a rare case. Lately I’ve been working a lot with extracts of cities and I get into these kinds of problems all the time: riverbanks, forests, administrative boundaries are typical examples of multipolygons that tend to be big and become ambiguous when used in extracts.

Basically, OSM multipolygons are a disaster. The “rules” (a.k.a. “recommendations”) on the wiki page change often, and not for the better. There’s a tendency to recommend very liberal ways of constructing multipolygons - nothing’s really mandatory and it is left to the consumer of the data to deal with the headaches. This is a very shortsighted approach, in my opinion, and it will hurt the project and the community in the (not so) long term. If the data cannot be used by “ordinary” users without complex toolchains and expensive hardware resources, then what is the purpose of it all?

Data hoarding for the sake of it?

8:07pm  |   URL: http://tmblr.co/Zm5DjxhZxiHS
(Notes: 6)
  
August 24, 2012

irbash asked: Hi, I was following your 10 steps, and I am stuck at the 9th one where I am trying to do ftp-upload. I get an error when I do that. I have created a local ftp site and I can access it from the browser, but when I give the same in maperitive I get a log list of error, the first one of which is: SystemAggregateException: One or more errors occurred SocketException: The requested name is valid, but no data of requested type was found. Can you help me with this issue... Thanks

Googling for “The requested name is valid, but no data of requested type was found” tells me it’s some network-related issue. Maybe your ftp-upload command line is wrong - make sure the host parameter points to a valid IP or host name (without the port number).

8:07pm  |   URL: http://tmblr.co/Zm5DjxS2sgzs
June 30, 2012
Maperitive: Alpenglow Effect Using A Custom Shader

The latest release comes with a new Python script called Alpenglow.py which renders a hillshading relief adding some yellowish tint to illuminated high slopes to enhance the 3D effect. It was inspired by the “See the light: How to make illuminated shaded relief in Photoshop 6.0” article on the excellent shadedrelief.com site.

You can run the script by executing

run-python Samples/Python/Alpenglow.py

The script generates the hillshading for a fixed map area in Switzerland, but you can change this quite easily in the script. You can also change other parameters, like shading (and alpenglow) colors, elevation ranges etc.

The script also demonstrates the power of writing a custom shader. A shader transforms a small part of the digital elevation model (DEM) into a pixel color, based on the elevation, aspect and the slope of that part of the DEM. This way you can write your own hillshading effects function and then supply it to the generator. Here’s how it’s done for Alpenglow.py:

# Execute the hillshading
bitmap = ReliefUtils.generate_relief_bitmap(dem, relief_bbox, Srid.WebMercator, 1, alpenglow)

The last parameter of the generate_relief_bitmap function is the shader function, in our case called alpenglow:

# The actual shader function which is called for each pixel of the hillshading bitmap.
# It receives the elevation, the aspect (orientation) of the slope and the steepnes (slope).
def alpenglow(elevation, aspect, slope):
    sin_slope = math.sin (slope)

    nx = math.cos(aspect) * sin_slope
    ny = math.sin(aspect) * sin_slope
    nz = math.cos(slope)

    direction = dotproduct(nx, ny, nz, lx, ly, lz)

    direction = (direction - minDirection) / (1 - minDirection);
    if direction < 0:
        direction = 0

    # This is how we can mix two colors together.
    shade_color = shade_base_color.mix (light_color, direction)

    if direction > 0.0 and elevation > min_alpenglow_elevation:
        elevation_factor = min((elevation - min_alpenglow_elevation) / delta_alpenglow_elevation, 1);

        alpenglow_factor = math.pow(direction, 4)
        return shade_color.mix (alpenglow_color, elevation_factor * alpenglow_factor).argb;

    return shade_color.argb;

6:22am  |   URL: http://tmblr.co/Zm5DjxOP5_BS
(Notes: 13)
  
Filed under: Maperitive maperipy python maps cartography hillshading 
May 24, 2012
Mercator’s Egg

UPDATE: my math was buggy, the eggs look a bit different after the correction. Note that the formula fails on very large distances (> 2000 km).

A little Maperitive Python script I was playing with today, showing the line of distance of 3000 km (larger egg) and 2000 km (smaller egg) from my home town of Maribor, drawn on the standard Web Mercator projection map:

Here’s the script:

from maperipy import *
import math

location = Point(15.6530555556, 46.5575)
distance = 3000000

layer = Map.add_custom_layer()

symbol = PolygonSymbol("circle", Srid.Wgs84LonLat)
symbol.style.pen_width = 8
symbol.style.pen_color = Color("blue")
symbol.style.pen_opacity = 0.5
symbol.style.fill_opacity = 0

earth_circumference = 6371000 * math.pi * 2;
points = []
arc = distance / earth_circumference * 360;

for angle in range(0, 360):
    radians = math.radians(angle)
    y = location.y + math.sin(radians) * arc
    lat_factor = math.cos(math.radians(y))
    x = location.x + math.cos(radians) * arc / lat_factor
    points.append(Point(x,y))

circle = Polygon(LinearRing(points))
symbol.add(circle)

layer.add_symbol(symbol)

4:40pm  |   URL: http://tmblr.co/Zm5DjxM2_3Zv
Filed under: Maperitive maperipy cartography maps 
May 16, 2012
11 Things You Should Learn Before Learning To Code

Here’s a list of ten skills that are much more important to get you through life than learning to code (in order of priority, set by me):

  1. Learn how to swim.
  2. Learn how to stay healthy and in shape.
  3. Learn the basics of food energy intake.
  4. Learn how to cook a decent (and healthy) meal.
  5. Learn how to use your mother tongue.
  6. Learn how to keep your personal finances in order. Think Greece & subprime loans.
  7. Learn a foreign language.
  8. Learn how to ride a bike.
  9. Learn how to drive a car.
  10. Learn how to use a washing machine.
  11. Learn how to find stuff on the internet.

8:07pm  |   URL: http://tmblr.co/Zm5DjxLbFnXs
May 5, 2012
Maperitive: Walking Papers-Like Script

The latest Maperitive 2.0 beta release comes with a new script which can generate a grid of bitmap exports similar to walking papers. The added value is that you can do walking papers for your own vector data and map styles, not just prerendered web tiles.

How To Use

A brief description of how to generate your own walking papers exports.

1. Define Map Area

In Maperitive, place printing bounds on the map area you’re interested in. The printing bounds do not have to match the dimensions of standard paper sizes.

I choose the center of Ljubljana for this sample:

2. Set Export Parameters

There are a few settings in the Samples/Python/WalkingPapers.py script which you can use to control the exporting process (see the detailed instructions in the script itself):

  • map scale: this is the map scale for individual exported page. I choose 10,000, which in my case will result in a 2x3 pages grid.
  • paper size and orientation: you can leave the default A4 landscape, if you like.
  • paper margins: the default is 5 mm on each side of the page.
  • page overlap: this is a factor (from 0 to 0.25) which tells Maperitive how much adjacent pages should overlap. 0.25 means 1/4 of the page (on each side) will overlap. The default set in the script is 0.1 (10%).

3. Run The Script

… either by drag & dropping it into Maperitive or by executing

 run-python Samples/Python/WalkingPapers.py

Sample Results

Here’s one of the exported pages:

Future Improvements

You will be able to supply command-line parameters to these kinds of scripts in the future, so manual editing of the script before running it will no longer be necessary.

9:17am  |   URL: http://tmblr.co/Zm5DjxKvDIYp
May 4, 2012
Oxford Bus & Cycle Map in print

I’ve just received the “Oxford Bus & Cycle Map” from Richard Mann (thanks, Richard!). The interesting thing is that it was made from OpenStreetMap data using Maperitive. It’s a stunning beauty, Richard has put a lot of effort into it.

Online versions of the maps are here: http://www.transportparadise.co.uk/maps/

1:35pm  |   URL: http://tmblr.co/Zm5DjxKrrHLL
Filed under: Maperitive cartography openstreetmap 
May 4, 2012
Results Of The Maperitive Survey

Last week I posted a quick survey for Maperitive users. The basic motivation was to see how many people are still using Windows XP SP2, so I can decide whether I can migrate Maperitive to .NET 4.0 (which is not available on SP2). But I added a few more questions in the mix, just for the fun of it.

I got 138 reponses so far, which is much more than I expected, so thanks for everyone who participated.

The results (I just copy&pasted charts from Google Docs, sorry for the labels cut-off):

What operating system(s) are you using to run Maperitive?

Participants could choose multiple answers - in fact about 1/4 of participants were using Maperitive on multiple operating systems. About 4% of people marked they are using Windows XP SP2 - I am wondering what is the reason they haven’t upgraded to SP3 (my guess is they are using Maperitive in corporate environments which tend to be slow on upgrades).

Two things that I think are important from these results:

  1. I can start work on migrating to .NET 4.0. For those unfortunate few who will be left out by this I can only say: try to upgrade to Service Pack 3. The official .NET 4.0 has been released two years ago (and first beta was released back in 2009) and it’s increasingly hard to support development on older versions of .NET framework. 3rd party libraries Maperitive depends on have mostly migrated to .NET 4.0 and this means missing out on new features and bugfixes.
  2. A lot (50%) of people are using Maperitive on Linux and Mac. This is much more than I expected and it means I will still have to keep Maperitive running on Mono. This is sometimes hard, since not everything is supported by Mono correctly. It also means a lot of people are willing to put up with an inferior user interface (due to Mono porting issues) for the sake of the functionality Maperitive offers. This is complimentary.

What it is the main reason you’re using Maperitive?

No surprises here, other than that at least 1/3 of users find Maperitive simple to use. This is a high number, given the fact that the documentation is not that good.

There’s one interesting (and a little bit surprising) answer:

We can use OSM offline, without prerendering thousands of tiles. It is very helpful for emergency services, because you cannot rely on a mobile internet connection everywhere or in the case of a disaster.

What kind of a user do you consider yourself to be?

Looks like the main audience of Maperitive are users who like to hack. Only 18% of participants in the survey consider themselves to be non-technical. I guess that explains the large “simple to use” percentage in the previous question :)

I use Maperitive mainly to…

Another question with multiple possible answers. I’m pleased that a lot of you are using Maperitive’s “power tools” like tile generation and exporting. I’m also pleased that at least 37% of participants are using the SVG export functionality, which I worked hard to implement.

I use Maperitive mainly for…

I guess I shouldn’t be surprised that most of you are doing hobby mapping with Maperitive. As OpenStreetMap reaches more people (and companies), I think the percentage of non-profit and commercial usages will increase.

Conclusion

The survey is still open (I think I will keep it open for some time), so you’re welcome to have your say. There will be more surveys in the future - this one was done as an experiment, in just 5 minutes. Again, thanks for participating!

10:41am  |   URL: http://tmblr.co/Zm5DjxKre3OP
Filed under: Maperitive survey 
April 29, 2012
Maperipy sample: generate tiles and then optimize with OptiPNG

The latest (2.0.28) Maperitive beta now comes with a sample Python script showing how you can override the default behavior of Maperitive tile generation process. The sample collects the generated tile files and then executes OptiPNG on them to reduce the file size (OptiPNG is now included in Maperitive package) of PNGs.

Some more info about this release: http://groups.google.com/group/maperitive/browse_thread/thread/492671b461f1e1da

Here’s the source for the sample, I think it’s self explanatory:

from maperipy import App
from maperipy import Map
from maperipy.webmaps import Tile
from maperipy.tilegen import TileGenCommand

def func1 (zoom, x, y, tiles_horizontal, tiles_vertical):
    # Demonstrates how to control which tiles get rendered
    # note that Maperitive generates tiles in batches
    # (defined by tiles_horizontal x tiles_vertical count).
    # So for example, 3x3 (=9) tiles are generated in a single bitmap 
    # and then later they are split into individual tiles.

    # In this case we just print some info to Maperitive's command log window.
    print "Generating tile batch: {0}/{1}/{2} ({3}x{4} tiles)" \
        .format(zoom, x, y, tiles_horizontal, tiles_vertical)

    # In this case we return True to signal that we want 
    # these tiles to be rendered.
    return True

def func2 (tile):
    # Demonstrates how to control the saving of a tile to disk.
    # At this point we could do some postprocessing of the tile bitmap
    # or store it somewhere else (and then return False to prevent
    # Maperitive from saving it to the default location).
    print "Saving tile: {0}/{1}/{2} (file: {3})" \
        .format(tile.zoom, tile.tile_x, tile.tile_y, tile.file_name)

    # You could, for example, store tiles into a database instead
    # of writing them to disk.

    # In this case we return True to signal that we want 
    # the tile to be saved to the default location.
    return True    

def optimize_tiles():
    args = []

    # Turn on backup files (just to compare sizes with optimized tiles).
    #args.append ("-backup")

    # Set the OptiPNG optimization rate (0 to 7), 7 being very slow.
    optimization_rate = 2
    args.append ("-o{0}".format(optimization_rate))

    # We don't want any OptiPNG output text to clutter Maperitive.
    args.append ("-quiet")

    args.extend(tiles_to_optimize)

    # This is one way to execute external programs.
    App.run_program("tools/OptiPNG/optipng.exe", 100, args)

    # Clear the list.
    del tiles_to_optimize[:]

def collect_tiles (file_name):
    # Demonstrates how to collect the list of generated tiles.
    # You could, for example, use this function to upload
    # tiles to a Web server.

    # In this case we collect the file names to a list...
    tiles_to_optimize.append(file_name)

    # ... and then we call OptiPNG when the list gets full.
    if len(tiles_to_optimize) >= 10:
        optimize_tiles()

tiles_to_optimize = []

# Generate tiles for the currently set geometry bounds and the current zoom level
# (Map.position.zoom, Map.position.zoom) -> min.zoom, max.zoom
cmd = TileGenCommand(Map.geo_bounds, Map.position.zoom, Map.position.zoom)

# This is how we override TileGenCommand with custom functions.
cmd.tile_generation_filter = func1
cmd.tile_save_filter = func2
cmd.after_tile_save = collect_tiles

cmd.execute()

# Run the optimization again for any left-over tiles.
optimize_tiles()

10:05am  |   URL: http://tmblr.co/Zm5DjxKXYDgR
Filed under: Maperitive maperipy Maperitive beta 
March 19, 2012
New OSM License, Some Practical Implications And Gray Areas

Occasionally I make digital maps for customers based on OpenStreetMap data. Sometimes they request the maps in form of bitmaps, and sometimes they need a vector format like SVG or PDF in order to be able to edit the maps in Adobe Illustrator. Of course, the issue of license always pops up and often have to explain the stipulations of CC BY-SA 2.0 to them.

Soon OSM will switch to a new ODbL license. I have to admit I mostly stayed away from the numerous legal talks that were going on various OSM channels, simply because I feel much more productive coding than participating in endless strings of emails. But now that the new license is here, I need to get acquainted with it from the perspective of someone trying to make (some) living out of OSM data. “I’m not a lawyer” is the usual phrase you can see in OSM legal discussions, but waiting for one to give you some solid information is like waiting for Godot, so I’ll make judgements based on my own understanding and some common sense instead, and simplify things when I feel like it. If anyone objects, they can twitter me with their objections and I’ll try to correct things.

So let’s say a customer requests an SVG map of my home town and I decide to use OSM data for it. For the sake of simplicity the map will be based purely on OSM data, so no other sources. Let’s first look at some of the important definitions in ODbL (emphases are mine):

“Database” – A collection of material (the Contents) arranged in a systematic or methodical way and individually accessible by electronic or other means offered under the terms of this License.

“Derivative Database” – Means a database based upon the Database, and includes any translation, adaptation, arrangement, modification, or any other alteration of the Database or of a Substantial part of the Contents. This includes, but is not limited to, Extracting or Re-utilising the whole or a Substantial part of the Contents in a new Database.

“Contents” – The contents of this Database, which includes the information, independent works, or other material collected into the Database. For example, the contents of the Database could be factual data or works such as images, audiovisual material, text, or sounds.

“Produced Work” – a work (such as an image, audiovisual material, text, or sounds) resulting from using the whole or a Substantial part of the Contents (via a search or other query) from this Database, a Derivative Database, or this Database as part of a Collective Database.

“Substantial” – Means substantial in terms of quantity or quality or a combination of both. The repeated and systematic Extraction or Re-utilisation of insubstantial parts of the Contents may amount to the Extraction or Re-utilisation of a Substantial part of the Contents.

So the first open question: is an SVG map a Produced Work or a Derivative Database? Or both? SVG map is an XML file that contains projected geographical data (together with visual styling attributes). OSM XML file can safely said to be a database. If you say SVG is not a database, where do you draw the line? What about KML or GML files?

This question is important because of the next clause:

Access to Derivative Databases. If You Publicly Use a Derivative Database or a Produced Work from a Derivative Database, You must also offer to recipients of the Derivative Database or Produced Work a copy in a machine readable form of:

a. The entire Derivative Database; or

b. A file containing all of the alterations made to the Database or the method of making the alterations to the Database (such as an algorithm), including any additional Contents, that make up all the differences between the Database and the Derivative Database.

If the SVG map file is not considered a Derivative Database, then you have an option of supplying the original OSM data (OSM XML file, PBF file or even a database snapshot) together with the SVG file or providing a description of how you derived the Derivative Database.

On the other hand, I can argue that SVG is a Derivative Database because it is “arranged in a systematic or methodical way and individually accessible” and “includes any translation, adaptation, arrangement, modification, or any other alteration” of the original OSM data. So in that case simply publishing the SVG file (and only that file) would cover the license requirements.

I should note that the SVG map has to be released under the ODbL or a compatible license.

Now let’s go one step further. Let’s assume (as I do) SVG is a Derivative Database. What if I then generate a PNG bitmap (or a Web map, for that matter) from the SVG file using Adobe Illustrator and want to publish that, too? One could argue that a bitmap is a Produced Work and since we already published the Derivative Database that produced this Work, we are covered.

But what if I didn’t generate the Web map from the SVG, but used a tool like Mapnik or Maperitive and generated it directly from an OSM extract instead? Let’s say that for practical purposes I don’t want to publish 1 GB of OSM data and I choose to go down the path of describing the “method of making the alterations” I did to generate the bitmap. What are the options here?

  1. I could write a detailed description of steps I performed to generate the Web map. Osmosis, Mapnik with all the batch scripts etc. I could even post the source code of the program(s) I used.
  2. On the other hand, I could just describe the process in a sentence or two. I could also say I used a special filter in Photoshop.

I can partly understand the spirit of the “method” clause - to enable access to the interesting derivations of the original data. But I see several holes in the “method” definition:

  1. What if I produced the map by arranging a lot of the map elements manually, by hand? This is quite a common case when you have to place map labels in order to avoid label conflicts. How would I describe the “method” other than saying that I did it by hand? How would that help anyone?
  2. What if I used an expensive proprietary software (like Illustrator or Photoshop)? Or even a piece of code that I haven’t released to anyone else? In that case nobody else would be able to reproduce the method. Does the “Contents” cover source code as well? It doesn’t mention the source code explicitly. If it does, then that implies you can only use open source software with OSM data, which would be silly.
  3. What about complex algorithms? How detailed the description would have to be for someone to be able to reproduce the algorithm? I’ve tried reproducing various algorithms from long scientific articles and I can tell you it’s not an easy task even if you have a detailed description.

Frankly, I don’t see how the “method” clause could be enforced in practice.

(UPDATE: now that I thought about it once more, the clause only talks about describing the method of arriving to the Derivative Database and not to the Produced Work itself. So I could just say “I downloaded the OSM extract from Geofabrik” and that would be it.)

One final question: does extracting OSM data for a city amount to a “substantial part” of the original OSM database?

8:40pm  |   URL: http://tmblr.co/Zm5DjxIFGTGz
(Notes: 2)
  
Filed under: osm openstreetmap ODbL licensing 
Liked posts on Tumblr: More liked posts »