OpenStreetMap: Difference between revisions

From Wildsong
Jump to navigationJump to search
Brian Wilson (talk | contribs)
mNo edit summary
Brian Wilson (talk | contribs)
mNo edit summary
 
(9 intermediate revisions by the same user not shown)
Line 1: Line 1:
Here is a page about 2 different ways to use PostGIS with OSM data.
== OpenStreetMap in a wiki ==
 
See my samples in [[Wiki Maps]].
 
== Controls ==
 
See ol-ext for a collection of fancy add on controls including a layerswitcher.
Here is the [http://viglino.github.io/ol-ext/ sample page] and here is the [http://viglino.github.io/ol-ext/doc/doc-pages/ API documentation].
 
== Editing in OSM ==
 
=== Using JOSM to edit ===
 
Use JOSM. It's a pain to learn YET ANOTHER GIS tool but it works.
 
See https://wiki.openstreetmap.org/wiki/JOSM/Plugins/OpenData
 
Use JOSM unless you are not using a Mac. It crashes on Linux
 
java -jar josm-tested.jar
 
=== Using GIS data in OSM ===
 
Data has to be in GPX format, so, after generating features in ArcMap (or QGIS),
 
# Save as a shapefile (from ArcMap)
# In QGIS,
## Load the ArcMap shapefile (or create some data)
## Convert multipart features to single part: Vector->Geometry->Multipart to Single part
## Convert polygon to line (also in Geometry tools)
## Right click layer and uses Save As
## Output to GPX, and select GPS_USE_EXTENSIONS = YES and FORCE_TRACK = YES
## Select TRACK when it asks what to save
 
Now you should be able to load the GPX file into ID by dropping into the browser window when editing.
This is not fun though because it puts a pink line on the screen and you have to then create the shape AGAIN. BOOOring!
 
== Using OSM data in PostGIS ==
 
Here are 2 different ways to use PostGIS with OSM data.
I list Osmosis first because it's what I am looking at right now.  
I list Osmosis first because it's what I am looking at right now.  
Then I list osm2pgsql which I found to be workable and easier than osmosis.
Then I list osm2pgsql which I found to be workable and easier than osmosis.
Line 5: Line 44:
I need to find out what the pros and cons are and list them here for you.  
I need to find out what the pros and cons are and list them here for you.  


=== Either way, first get the data ===
Get the data for the western United States, grab the PBF either way.
wget http://download.geofabrik.de/north-america/us-west-latest.osm.pbf


== Using osmosis ==
=== Using osmosis ===


Let the data seep into PostGIS through a semipermeable membrane with osmosis, or use the command line tool of the same name.
Let the data seep into PostGIS through a semipermeable membrane with osmosis, or use the command line tool of the same name.


=== Get the tool ===
==== Get the tool ====


This did not work very well for me
This did not work very well for me
Line 24: Line 68:
  ./gradlew assemble
  ./gradlew assemble


=== Set up the database ===
==== Set up the database ====


  createdb osmosis_us_west
  createdb osm_us_west
  echo "CREATE EXTENSION hstore; CREATE EXTENSION postgis;" | psql -U postgres $DB
  echo "CREATE EXTENSION hstore; CREATE EXTENSION postgis;" | psql -U postgres $DB
  echo "ALTER TABLE osmosis_us_west OWNER TO osm" | psql -U postgres
  echo "ALTER TABLE osm_us_west OWNER TO osm" | psql -U postgres
  cd script
  cd script
  psql -U postgres osmosis_us_west -f pgsnapshot_schema_0.6.sql  
  psql -U postgres osm_us_west -f pgsnapshot_schema_0.6.sql  
  psql -U postgres osmosis_us_west -f pgsnapshot_schema_0.6_action.sql  
  psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_action.sql  
  psql -U postgres osmosis_us_west -f pgsnapshot_schema_0.6_bbox.sql  
  psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_bbox.sql  
  psql -U postgres osmosis_us_west -f pgsnapshot_schema_0.6_linestring.sql  
  psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_linestring.sql


=== Get the data ===
==== Load the data into PostGIS ====
 
Get the data for the western United States
wget http://download.geofabrik.de/north-america/us-west-latest.osm.pbf


Run osmosis to load the PBF file into your PostGIS server. I do it thusly
Run osmosis to load the PBF file into your PostGIS server. I do it thusly
Line 49: Line 90:
At this point I have the data loaded and to prove that it's usable I need to try QGIS to view it and I need to extract a subset.
At this point I have the data loaded and to prove that it's usable I need to try QGIS to view it and I need to extract a subset.


== Using osm2pgsql tool ==
=== Using osm2pgsql tool ===


Using excellent instructions from http://bostongis.com/
Using excellent instructions from http://bostongis.com/
In particular, the [http://bostongis.com/?content_name=loading_osm_postgis#229 Almost idiot's guide...]
In particular, the [http://bostongis.com/?content_name=loading_osm_postgis#229 Almost idiot's guide...]


First, get the data from CloudMade for California and Oregon, or if you have LOTS of time and space get the whole planet.
First, get the data as above, the PBF file for western US.
 
mkdir /green/GISData/OSM && cd /green/GISData/OSM
wget http://downloads.cloudmade.com/americas/northern_america/united_states/california/california.osm.bz2
wget http://downloads.cloudmade.com/americas/northern_america/united_states/oregon/oregon.osm.bz2
 
Create a lovely Postgis database and spatially enable it and add hstore.
Create a lovely Postgis database and spatially enable it and add hstore.


=== Build osm2pgsql ===
==== Build osm2pgsql ====


No need for any special configuration options, just build and install!
No need for any special configuration options, just build and install!


  sudo apt-get install libxml2-dev libbz2-dev
  sudo apt-get install cmake libboost-dev libboost-system-dev libboost-filesystem-dev zlib1g-dev libbz2-dev lua5.2 liblua5.2-dev
  cd ~/src/GIS
  cd ~/src/GIS
  git clone https://github.com/openstreetmap/osm2pgsql.git
  git clone https://github.com/openstreetmap/osm2pgsql.git
  cd osm2pgsql
  cd osm2pgsql
  ./autogen.sh
  mkdir build
  ./configure
  cd build
  sed -i 's/-g -O2/-O2 -march=native -fomit-frame-pointer/' Makefile
  cmake ..
  make
  make
  sudo make install
  sudo make install


=== Using osm2pgsql ===
==== Using osm2pgsql ====


  # bellman
  # bellman
Line 83: Line 119:


  # dart
  # dart
createuser -U postgres osm
  createdb -U postgres osm
  createdb -U postgres osm
createuser -U postgres osm
  psql -U postgres osm  
  psql -U postgres osm  
  CREATE EXTENSION hstore;
  CREATE EXTENSION hstore;
Line 90: Line 126:
   
   
  # The western US - I had to increase the node cache size
  # The western US - I had to increase the node cache size
  osm2pgsql us-west-latest.osm.pbf -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C 10000
  osm2pgsql us-west-latest.osm.pbf \
  -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C 10000 \
  --number-processes 8 --multi-geometry
 
psql -U postgres osm
  CREATE INDEX idx_planet_osm_point_tags ON planet_osm_point USING gist(tags);
  CREATE INDEX idx_planet_osm_polygon_tags ON planet_osm_polygon USING gist(tags);
  CREATE INDEX idx_planet_osm_line_tags ON planet_osm_line USING gist(tags);


  # The whole planet - I RUN OUT OF MEMORY!! Dang.
  # The whole planet - I RUN OUT OF MEMORY!! Dang.
Line 102: Line 145:


Now I need to get it showing up in [[Geoserver]]
Now I need to get it showing up in [[Geoserver]]
[[Category: GIS]]

Latest revision as of 21:10, 8 October 2018

OpenStreetMap in a wiki

See my samples in Wiki Maps.

Controls

See ol-ext for a collection of fancy add on controls including a layerswitcher. Here is the sample page and here is the API documentation.

Editing in OSM

Using JOSM to edit

Use JOSM. It's a pain to learn YET ANOTHER GIS tool but it works.

See https://wiki.openstreetmap.org/wiki/JOSM/Plugins/OpenData

Use JOSM unless you are not using a Mac. It crashes on Linux

java -jar josm-tested.jar

Using GIS data in OSM

Data has to be in GPX format, so, after generating features in ArcMap (or QGIS),

  1. Save as a shapefile (from ArcMap)
  2. In QGIS,
    1. Load the ArcMap shapefile (or create some data)
    2. Convert multipart features to single part: Vector->Geometry->Multipart to Single part
    3. Convert polygon to line (also in Geometry tools)
    4. Right click layer and uses Save As
    5. Output to GPX, and select GPS_USE_EXTENSIONS = YES and FORCE_TRACK = YES
    6. Select TRACK when it asks what to save

Now you should be able to load the GPX file into ID by dropping into the browser window when editing. This is not fun though because it puts a pink line on the screen and you have to then create the shape AGAIN. BOOOring!

Using OSM data in PostGIS

Here are 2 different ways to use PostGIS with OSM data. I list Osmosis first because it's what I am looking at right now. Then I list osm2pgsql which I found to be workable and easier than osmosis.

I need to find out what the pros and cons are and list them here for you.

Either way, first get the data

Get the data for the western United States, grab the PBF either way.

wget http://download.geofabrik.de/north-america/us-west-latest.osm.pbf

Using osmosis

Let the data seep into PostGIS through a semipermeable membrane with osmosis, or use the command line tool of the same name.

Get the tool

This did not work very well for me No build required, that's the Java JAR advantage! (You knew there was an advantage.)

wget http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip
mkdir osmosis
cd osmosis
unzip ../osmosis-latest.zip

This is what I ended up doing

git clone https://github.com/openstreetmap/osmosis.git
cd osmosis
./gradlew assemble

Set up the database

createdb osm_us_west
echo "CREATE EXTENSION hstore; CREATE EXTENSION postgis;" | psql -U postgres $DB
echo "ALTER TABLE osm_us_west OWNER TO osm" | psql -U postgres
cd script
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6.sql 
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_action.sql 
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_bbox.sql 
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_linestring.sql

Load the data into PostGIS

Run osmosis to load the PBF file into your PostGIS server. I do it thusly

# Put TEMP files somewhere with lots of space
mkdir tmp
JAVACMD_OPTIONS="-Djava.io.tmpdir=tmp" \
./osmosis -v --read-pbf us-west-latest.osm.pbf --write-pgsql user=osm_reader database=osmosis_us_west

At this point I have the data loaded and to prove that it's usable I need to try QGIS to view it and I need to extract a subset.

Using osm2pgsql tool

Using excellent instructions from http://bostongis.com/ In particular, the Almost idiot's guide...

First, get the data as above, the PBF file for western US. Create a lovely Postgis database and spatially enable it and add hstore.

Build osm2pgsql

No need for any special configuration options, just build and install!

sudo apt-get install cmake libboost-dev libboost-system-dev libboost-filesystem-dev zlib1g-dev libbz2-dev lua5.2 liblua5.2-dev
cd ~/src/GIS
git clone https://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql
mkdir build
cd build
cmake ..
make
sudo make install

Using osm2pgsql

# bellman
osm2pgsql oregon.osm.bz2 -d osm -U postgres -S /green/bwilson/src/GIS/osm2pgsql/default.style --hstore
osm2pgsql california.osm.bz2 --append -d osm -U postgres -S /green/bwilson/src/GIS/osm2pgsql/default.style --hstore
# dart
createuser -U postgres osm
createdb -U postgres osm
psql -U postgres osm 
CREATE EXTENSION hstore;
CREATE EXTENSION postgis;

# The western US - I had to increase the node cache size
osm2pgsql us-west-latest.osm.pbf \
  -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C 10000 \
  --number-processes 8 --multi-geometry
psql -U postgres osm 
 CREATE INDEX idx_planet_osm_point_tags ON planet_osm_point USING gist(tags);
 CREATE INDEX idx_planet_osm_polygon_tags ON planet_osm_polygon USING gist(tags);
 CREATE INDEX idx_planet_osm_line_tags ON planet_osm_line USING gist(tags);
# The whole planet - I RUN OUT OF MEMORY!! Dang.
osm2pgsql planet-latest.osm -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C 15000

On my little server Bellman, it took 628 seconds to load the Oregon data and it took 3302 seconds to load California. Conclusion: California is much bigger, in fact there is probably room for me there in Nevada county.

I just loaded Oregon data onto my much snazzier server at work and it took 183 seconds.

Nice to have some data in my PostGIS server.

Now I need to get it showing up in Geoserver