OpenStreetMap: Difference between revisions

From Wildsong
Jump to navigationJump to search
Brian Wilson (talk | contribs)
Brian Wilson (talk | contribs)
mNo edit summary
 
(21 intermediate revisions by the same user not shown)
Line 1: Line 1:
Using excellent instructions from http://bostongis.com/
== OpenStreetMap in a wiki ==
In particular, the [http://bostongis.com/?content_name=loading_osm_postgis#229 Almost idiot's guide...]
 
See my samples in [[Wiki Maps]].
 
== Controls ==
 
See ol-ext for a collection of fancy add on controls including a layerswitcher.
Here is the [http://viglino.github.io/ol-ext/ sample page] and here is the [http://viglino.github.io/ol-ext/doc/doc-pages/ API documentation].
 
== Editing in OSM ==
 
=== Using JOSM to edit ===
 
Use JOSM. It's a pain to learn YET ANOTHER GIS tool but it works.
 
See https://wiki.openstreetmap.org/wiki/JOSM/Plugins/OpenData
 
Use JOSM unless you are not using a Mac. It crashes on Linux
 
java -jar josm-tested.jar
 
=== Using GIS data in OSM ===
 
Data has to be in GPX format, so, after generating features in ArcMap (or QGIS),
 
# Save as a shapefile (from ArcMap)
# In QGIS,  
## Load the ArcMap shapefile (or create some data)
## Convert multipart features to single part: Vector->Geometry->Multipart to Single part
## Convert polygon to line (also in Geometry tools)
## Right click layer and uses Save As
## Output to GPX, and select GPS_USE_EXTENSIONS = YES and FORCE_TRACK = YES
## Select TRACK when it asks what to save
 
Now you should be able to load the GPX file into ID by dropping into the browser window when editing.
This is not fun though because it puts a pink line on the screen and you have to then create the shape AGAIN. BOOOring!
 
== Using OSM data in PostGIS ==
 
Here are 2 different ways to use PostGIS with OSM data.
I list Osmosis first because it's what I am looking at right now.
Then I list osm2pgsql which I found to be workable and easier than osmosis.
 
I need to find out what the pros and cons are and list them here for you.
 
=== Either way, first get the data ===
 
Get the data for the western United States, grab the PBF either way.
 
wget http://download.geofabrik.de/north-america/us-west-latest.osm.pbf
 
=== Using osmosis ===
 
Let the data seep into PostGIS through a semipermeable membrane with osmosis, or use the command line tool of the same name.
 
==== Get the tool ====
 
This did not work very well for me
No build required, that's the Java JAR advantage! (You knew there was an advantage.)
wget http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip
mkdir osmosis
cd osmosis
unzip ../osmosis-latest.zip


First, get the data from CloudMade for California and Oregon
This is what I ended up doing
git clone https://github.com/openstreetmap/osmosis.git
cd osmosis
./gradlew assemble


mkdir /green/GISData/OSM && cd /green/GISData/OSM
==== Set up the database ====
wget http://downloads.cloudmade.com/americas/northern_america/united_states/california/california.osm.bz2
wget http://downloads.cloudmade.com/americas/northern_america/united_states/oregon/oregon.osm.bz2


Create a lovely Postgis database
createdb osm_us_west
echo "CREATE EXTENSION hstore; CREATE EXTENSION postgis;" | psql -U postgres $DB
echo "ALTER TABLE osm_us_west OWNER TO osm" | psql -U postgres
cd script
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6.sql
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_action.sql
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_bbox.sql
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_linestring.sql


'''createdb -U postgres osm -W -T template_postgis'''
==== Load the data into PostGIS ====


My installed-from-source Postgis did not have hstore.sql so I built it:
Run osmosis to load the PBF file into your PostGIS server. I do it thusly


  cd ~/src/GIS/postgresql-9.0.6/contrib/hstore
  # Put TEMP files somewhere with lots of space
USE_PGXS=1 make && sudo USE_PGXS=1 make install
mkdir tmp
  cd /usr/local/pgsql/share/contrib
JAVACMD_OPTIONS="-Djava.io.tmpdir=tmp" \
  ./osmosis -v --read-pbf us-west-latest.osm.pbf --write-pgsql user=osm_reader database=osmosis_us_west


Do this each time you create a database a: add hstore support to the database
At this point I have the data loaded and to prove that it's usable I need to try QGIS to view it and I need to extract a subset.


'''psql -U postgres osm < hstore.sql'''
=== Using osm2pgsql tool ===


=== Build osm2pgsql ===
Using excellent instructions from http://bostongis.com/
In particular, the [http://bostongis.com/?content_name=loading_osm_postgis#229 Almost idiot's guide...]
 
First, get the data as above, the PBF file for western US.
Create a lovely Postgis database and spatially enable it and add hstore.
 
==== Build osm2pgsql ====


No need for any special configuration options, just build and install!
No need for any special configuration options, just build and install!


  sudo apt-get install libxml2-dev libbz2-dev
  sudo apt-get install cmake libboost-dev libboost-system-dev libboost-filesystem-dev zlib1g-dev libbz2-dev lua5.2 liblua5.2-dev
  cd ~/src/GIS
  cd ~/src/GIS
  git clone https://github.com/openstreetmap/osm2pgsql.git
  git clone https://github.com/openstreetmap/osm2pgsql.git
  cd osm2pgsql
  cd osm2pgsql
  ./autogen.sh
  mkdir build
  ./configure
  cd build
  sed -i 's/-g -O2/-O2 -march=native -fomit-frame-pointer/' Makefile
  cmake ..
  make
  make
  sudo make install
  sudo make install


=== Using osm2pgsql ===
==== Using osm2pgsql ====


  # bellman
  # bellman
  osm2pgsql oregon.osm.bz2 -d osm -U postgres \
  osm2pgsql oregon.osm.bz2 -d osm -U postgres -S /green/bwilson/src/GIS/osm2pgsql/default.style --hstore
-S /green/bwilson/src/GIS/osm2pgsql/default.style --hstore
  osm2pgsql california.osm.bz2 --append -d osm -U postgres -S /green/bwilson/src/GIS/osm2pgsql/default.style --hstore
  osm2pgsql california.osm.bz2 --append -d osm -U postgres \
-S /green/bwilson/src/GIS/osm2pgsql/default.style --hstore


  # dart
  # dart
createuser -U postgres osm
  createdb -U postgres osm
  createdb -U postgres osm
createuser -U postgres osm
  psql -U postgres osm  
  psql -U postgres osm  
  CREATE EXTENSION hstore;
  CREATE EXTENSION hstore;
  CREATE EXTENSION postgis;
  CREATE EXTENSION postgis;
# Just Oregon - I had to increase the node cache size on Dart where I am loading US West not just Oregon. -C 3000
  osm2pgsql us-west-latest.osm.pbf -d osm -U osm -S /home/AGI/bwilson/src/GIS/osm2pgsql/default.style --hstore -C 3000
# The western US - I had to increase the node cache size
# The whole planet
  osm2pgsql us-west-latest.osm.pbf \
  osm2pgsql planet-latest.osm -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C 5000
  -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C 10000 \
  --number-processes 8 --multi-geometry
 
psql -U postgres osm
  CREATE INDEX idx_planet_osm_point_tags ON planet_osm_point USING gist(tags);
  CREATE INDEX idx_planet_osm_polygon_tags ON planet_osm_polygon USING gist(tags);
  CREATE INDEX idx_planet_osm_line_tags ON planet_osm_line USING gist(tags);
 
# The whole planet - I RUN OUT OF MEMORY!! Dang.
  osm2pgsql planet-latest.osm -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C '''15000'''


On my little server [[Bellman]], it took 628 seconds to load the Oregon data and it took 3302 seconds to load California. Conclusion: California is much bigger, in fact there is probably room for me there in Nevada county.
On my little server [[Bellman]], it took 628 seconds to load the Oregon data and it took 3302 seconds to load California. Conclusion: California is much bigger, in fact there is probably room for me there in Nevada county.
Line 62: Line 145:


Now I need to get it showing up in [[Geoserver]]
Now I need to get it showing up in [[Geoserver]]
[[Category: GIS]]

Latest revision as of 21:10, 8 October 2018

OpenStreetMap in a wiki

See my samples in Wiki Maps.

Controls

See ol-ext for a collection of fancy add on controls including a layerswitcher. Here is the sample page and here is the API documentation.

Editing in OSM

Using JOSM to edit

Use JOSM. It's a pain to learn YET ANOTHER GIS tool but it works.

See https://wiki.openstreetmap.org/wiki/JOSM/Plugins/OpenData

Use JOSM unless you are not using a Mac. It crashes on Linux

java -jar josm-tested.jar

Using GIS data in OSM

Data has to be in GPX format, so, after generating features in ArcMap (or QGIS),

  1. Save as a shapefile (from ArcMap)
  2. In QGIS,
    1. Load the ArcMap shapefile (or create some data)
    2. Convert multipart features to single part: Vector->Geometry->Multipart to Single part
    3. Convert polygon to line (also in Geometry tools)
    4. Right click layer and uses Save As
    5. Output to GPX, and select GPS_USE_EXTENSIONS = YES and FORCE_TRACK = YES
    6. Select TRACK when it asks what to save

Now you should be able to load the GPX file into ID by dropping into the browser window when editing. This is not fun though because it puts a pink line on the screen and you have to then create the shape AGAIN. BOOOring!

Using OSM data in PostGIS

Here are 2 different ways to use PostGIS with OSM data. I list Osmosis first because it's what I am looking at right now. Then I list osm2pgsql which I found to be workable and easier than osmosis.

I need to find out what the pros and cons are and list them here for you.

Either way, first get the data

Get the data for the western United States, grab the PBF either way.

wget http://download.geofabrik.de/north-america/us-west-latest.osm.pbf

Using osmosis

Let the data seep into PostGIS through a semipermeable membrane with osmosis, or use the command line tool of the same name.

Get the tool

This did not work very well for me No build required, that's the Java JAR advantage! (You knew there was an advantage.)

wget http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip
mkdir osmosis
cd osmosis
unzip ../osmosis-latest.zip

This is what I ended up doing

git clone https://github.com/openstreetmap/osmosis.git
cd osmosis
./gradlew assemble

Set up the database

createdb osm_us_west
echo "CREATE EXTENSION hstore; CREATE EXTENSION postgis;" | psql -U postgres $DB
echo "ALTER TABLE osm_us_west OWNER TO osm" | psql -U postgres
cd script
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6.sql 
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_action.sql 
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_bbox.sql 
psql -U postgres osm_us_west -f pgsnapshot_schema_0.6_linestring.sql

Load the data into PostGIS

Run osmosis to load the PBF file into your PostGIS server. I do it thusly

# Put TEMP files somewhere with lots of space
mkdir tmp
JAVACMD_OPTIONS="-Djava.io.tmpdir=tmp" \
./osmosis -v --read-pbf us-west-latest.osm.pbf --write-pgsql user=osm_reader database=osmosis_us_west

At this point I have the data loaded and to prove that it's usable I need to try QGIS to view it and I need to extract a subset.

Using osm2pgsql tool

Using excellent instructions from http://bostongis.com/ In particular, the Almost idiot's guide...

First, get the data as above, the PBF file for western US. Create a lovely Postgis database and spatially enable it and add hstore.

Build osm2pgsql

No need for any special configuration options, just build and install!

sudo apt-get install cmake libboost-dev libboost-system-dev libboost-filesystem-dev zlib1g-dev libbz2-dev lua5.2 liblua5.2-dev
cd ~/src/GIS
git clone https://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql
mkdir build
cd build
cmake ..
make
sudo make install

Using osm2pgsql

# bellman
osm2pgsql oregon.osm.bz2 -d osm -U postgres -S /green/bwilson/src/GIS/osm2pgsql/default.style --hstore
osm2pgsql california.osm.bz2 --append -d osm -U postgres -S /green/bwilson/src/GIS/osm2pgsql/default.style --hstore
# dart
createuser -U postgres osm
createdb -U postgres osm
psql -U postgres osm 
CREATE EXTENSION hstore;
CREATE EXTENSION postgis;

# The western US - I had to increase the node cache size
osm2pgsql us-west-latest.osm.pbf \
  -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C 10000 \
  --number-processes 8 --multi-geometry
psql -U postgres osm 
 CREATE INDEX idx_planet_osm_point_tags ON planet_osm_point USING gist(tags);
 CREATE INDEX idx_planet_osm_polygon_tags ON planet_osm_polygon USING gist(tags);
 CREATE INDEX idx_planet_osm_line_tags ON planet_osm_line USING gist(tags);
# The whole planet - I RUN OUT OF MEMORY!! Dang.
osm2pgsql planet-latest.osm -d osm -U osm -S ~/src/GIS/osm2pgsql/default.style --hstore -C 15000

On my little server Bellman, it took 628 seconds to load the Oregon data and it took 3302 seconds to load California. Conclusion: California is much bigger, in fact there is probably room for me there in Nevada county.

I just loaded Oregon data onto my much snazzier server at work and it took 183 seconds.

Nice to have some data in my PostGIS server.

Now I need to get it showing up in Geoserver