GeoServer: Difference between revisions

From Wildsong
Jump to navigationJump to search
← Older editNewer edit →
VisualWikitext
Brian Wilson (talk | contribs)
Bureaucrats, restrict, Administrators, uploaders
11,535 edits
mNo edit summary
Brian Wilson (talk | contribs)
Bureaucrats, restrict, Administrators, uploaders
11,535 edits
Line 175: Line 175:
=== Compose file ===
=== Compose file ===


Here is my Docker Compose file, so far.
Okay now that I have tested Geoserver, PostGIS, PostgreSQL, and pgadmin separately I am ready to roll them together with one Docker Compose file. Here is my file "docker-compose.yml", so far.
 
<pre>
<pre>
# Match this to your docker engine, I am at 17.06 in Debian 9
# Match this to your docker engine, I am at 18.09.02 in Debian 9
version: '3.3'
version: '3.7'


services:
services:
   db:
   db:
     image: camptocamp/postgres:latest
     image: mdillion/postgis:11
     environment:
     environment:
      PGDATA: /var/lib/postgresql/data
       POSTGRES_PASSWORD: example
       POSTGRES_PASSWORD: example
       #POSTGRES_USER:
       #POSTGRES_USER:
      #PGDATA:
       #POSTGRES_DB:
       #POSTGRES_DB:
       #POSTGRES_INITDB_ARGS:
       #POSTGRES_INITDB_ARGS:
       #POSTGRES_INITDB_XLOGDIR:
       #POSTGRES_INITDB_XLOGDIR:
     ports:
     ports:
     - "5433:5432"
     - "5432:5432"
     volumes:
     volumes:
     - "/var/lib/postgresql/docker:/var/lib/postgresql"
     - "postgis_files:/var/lib/postgresql/data"
 
    # I will want this behind an nginx reverse proxy instead of exposing the port
  # I will want this behind an nginx reverse proxy instead of exposing the port
 
   geoserver:
   geoserver:
     image: geoceg/geoserver:latest
     image: geoceg/geoserver:latest
     ports:
     ports:
     - "8080:8080"
     - "8888:8080"
 
  pgadmin:
    image: pgadmin
    ports:
    - ""
</pre>
</pre>


Line 229: Line 234:
==== Plumbing the network ====
==== Plumbing the network ====


Create a network to hook together the geoserver and postgis containers
Create a network to hook together the containers


  docker network create pg
  docker network create pg
  docker run --name postgis --net=pg -p 5432 -d wildsong/postgis
  docker run --name postgis --net=pg -p 5432 -d mdillon/postgis
  docker run --name geoserver --net=pg -p 8888:8080 -d wildsong/geoserver
  docker run --name geoserver --net=pg -p 8888:8080 -d geoceg/geoserver
  docker network inspect pg
  docker network inspect pg


The two containers can see each other now with the names 'postgis' and 'geoserver' resolving correctly.
The two containers can see each other now with the names 'postgis' and 'geoserver' resolving correctly.
The 'inspect' command shows the two containers are connected to the pg network.
The 'inspect' command shows the two containers are connected to the pg network.
=== Persisting the data ===
Add to the Dockerfile for PostGIS, at the end:
ENV PGDATA /pgdata
VOLUME ${PGDATA}
The ENV statement will tell Postgres you want to keep all its files in a place called /pgdata.
The VOLUME command allows you to tie that place "/pgdata" to a spot on your host computer when you run the container.
Change the run command by added a mapping between the host and /pgdata/, for example,
docker run --name postgis -v /home/myfavoriteplace/pgdata:/pgdata/ '''etc'''
will cause the files to be written to /home/myfavoriteplace/pgdata. This gives you the flexibility to use the same Docker image
on different computers, keeping the data stored in the local filesystem, but in a different place on each computer.


=== Dealing with passwords ===
=== Dealing with passwords ===
Line 259: Line 248:
I don't want passwords to be coded into my Docker images, so how do I get them out of there?
I don't want passwords to be coded into my Docker images, so how do I get them out of there?
I know I am supposed to put them into environment settings but then how do I load the environment?
I know I am supposed to put them into environment settings but then how do I load the environment?
=== Getting a custom docker onto a Synology ===
I can do this by pushing my custom built Docker image into the Docker Hub
and then pull it onto the Synology. I have set up both an organizational account
and a personal account. I log in with my personal credentials and then push to the organizational account.
docker login -u=brian32768 -p=''paswword''
docker push wildsong/postgis
Once the image is pushed then I should be able to pull it onto the Synology.
Passing everything through a repository some 1000's of miles away might not work for you; you could set up your own registry.
From the command line was easiest for me,
sudo docker pull geoceg/postgis
I used the web interface to create a container and launch it once the image was copied to the server.


== Setting up GeoServer on Linux ==
== Setting up GeoServer on Linux ==

Revision as of 22:16, 17 February 2019

Overview

GeoServer is what I used with PostgreSQL and PostGIS for the Mapping Vietnam project.

Home page: http://geoserver.org/ where they say "GeoServer is a Java-based software server that allows users to view and edit geospatial data. Using open standards set forth by the Open Geospatial Consortium (OGC), GeoServer allows for great flexibility in map creation and data sharing."

The crucial feature for me is that it allows edits via WFS-T. That means I can hook up OpenLayers web clients to a database and allow them to create and edit data.

WFS-T comments: http://weblogs.java.net/blog/2006/01/09/geoserver-wfs-t

History

  • 2019 Feb -- Finishing the migration to Docker Compose.
  • 2017 Sep -- Setting up Docker Compose version
  • 2016 Sep -- updated Dart with 2.9.1 (latest stable release)
  • 2012 Sep -- 2.2 installed on Kilchis and Dart.
  • 2011 Oct -- I am working with 2.1.2 now on HuPI
  • 2009 Feb 06 -- I decided to take a crack at installing GeoServer 1.7.2 today.

Example GeoServer site (not mine!)

The Portland TriMet system map is on GeoServer via the OpenGeo stack.

This is from the Portland Trimet site... GeoServer, OpenLayers, TileCache, PostgreSQL, PostGIS, GDAL, Extjs, Freemarker, Solr, Antlr, Balance Ant, Http Client, Tomcat, Apache, and CentOS are the underlying technologies used on this application. We're indebted to the contributors of these (and many other: eclipse, firefox and firebug to name a few) open source projects for sharing their efforts and expertise.

Some GeoServer Capabilities

  • WCS = Web Coverage Service = queryable feature data (OGC)
  • WFS = Web Feature Service = vector data (OGC)
  • WMS = Web Map Service = delivers rasters (not tiles) (OGC)
  • TMS = Tile map service = delivers raster tiles (OSGEO)
  • WMS-C = Web Map Service - Cached = WMS with tile caching (OSGEO draft)
  • WMTS = Web Map Tile service (OGC)

I think WCS can return features that are related from several feature classes whereas WFS returns features from one class only??

Map vs feature services

What if you want features to overlay a map but want symbology defined by the server? You define styles and then pick the correct encoding.

With WFS there are several GML flavours, GeoJSON, shapefiles and CSV. None of these work for me because they only encode the geography and not the visual properties. I probably want to use a WMS encoded with KML.

Lew's approach is to convert the KML into a custom JSON format on the server for transfer. There must be a better way than that. But it has to be compatible with the Google Maps Android API too.

The official way to put markers and shapes onto a Google map is via their API. I don't want to go the route of putting shapes onto the screen ignoring the API because I hate having to do all the projection and transformation stuff on my own.

Being able to directly read GPX files would be a good thing.

Tiles

If you are implementing a tile-based client you should probably use TMS or maybe WMTS. WMS-C was a OSGEO draft. WMTS is more complex so use TMS unless you really need WMTS features.

On the server side for performance and the best cartography you'd probably be better off pre-rendering and caching all the tiles. What's the best way to do that? Tilemill or gdal2tiles ? What's the best way to serve the prerendered tiles? TileCache? Or just use a web server (Apache)?

TMS reference implementation: see http://TileCache.org/

Building the Geo-CEG Geoserver Docker images

There will be 4 Docker containers in the finished project, pulled together with Compose.

  1. mdillon/postgis@latest
  2. geoceg/postgres will contain PostgreSQL with the ESRI SDE additions to make it compatible with ArcGIS
  3. geoceg/geoserver which contains geoserver and geowebcache
  4. dpage/pgadmin4

Outside of Docker I will use Debian nginx to proxy these services and hide behind its TLS encryption with proper certificates for https://maps.wildsong.biz and https://bellman.wildsong.biz

Geoserver

I created a docker image and put it on github, https://github.com/Geo-CEG/docker-geoserver and it autobuilds an image at https://hub.docker.com/r/geoceg/geoserver/ and I continue to refine and update that image as needed in this project.

Test it independently with 

docker run -t -p 8888:8080 --name=geoserver -v geoserver_files:/geoserver geoceg/geoserver:latest

I proxy port 8888 through nginx on Bellman so that it shows up as https://maps.wildsong.biz/geoserver/web/

Because it's tucked behind this proxy, it should be accessible everywhere (based on my Firewall settings but that's a different story).

Upgrade

I just updated the Github project from 2.13.2 to 2.14.2 and want to upgrade my installed image. 

docker stop geoserver
docker rm geoserver
docker pull geoceg/geoserver:latest

Then issue the same Docker run (or compose) used before.

PostGIS

See https://hub.docker.com/r/mdillon/postgis

See also its parent https://hub.docker.com/_/postgres/

These commands put the data and config files outside of the container. 

docker pull mdillon/postgis:11
docker volume create postgis_files

docker run -d --name postgis -v postgis_files:/var/lib/postgresql/data \
  -e PGDATA=/var/lib/postgresql/data -e POSTGRES_PASSWORD=secret mdillon/postgis:11

Run psql to talk to the database; 

docker exec -it postgis /usr/bin/psql -U postgres

This command actually starts a second container and links to the first one to do the same thing - give you psql. 

docker run -it --link postgis:postgres --rm postgres \
   sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres -W'

Either way once you get a prompt, you can do the quick test to see what's installed, 

postgres=# SELECT postgis_full_version();

POSTGIS="2.5.1 r17027" [EXTENSION] PGSQL="110" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016" 
GDAL="GDAL 2.1.2, released 2016/10/24" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" TOPOLOGY RASTER
(1 row)

Docker makes setting up PostGIS shockingly easy. You'll want pgadmin to maintain it.

ArcGIS SDE in PostgreSQL

Requirements: https://desktop.arcgis.com/en/system-requirements/latest/database-requirements-postgresql.htm It wants Ubuntu 16.04.3 PostgreSQL 10.3 and PostGIS 2.4

That's correct, it REQUIRES PostGIS. This hurts my brain.

This tells me to install a DLL which is stupid since I am running on Linux https://desktop.arcgis.com/en/arcmap/latest/manage-data/gdbs-in-postgresql/get-started-gdb-in-postgresql.htm

I think I need this https://desktop.arcgis.com/en/arcmap/latest/manage-data/databases/add-the-st-geometry-type-to-a-postgresql-database.htm

pgadmin

pgadmin used to be a desktop app, now it's a javascript app that can run still from desktop but I have it installed as a docker on Bellman. 

sudo mkdir /var/lib/docker/pgadmin

docker pull dpage/pgadmin4
docker volume create pgadmin_files



docker run -it --link postgis:postgres --rm postgres \

docker run -d -p 8889:80 --name=pgadmin4 -v pgadmin_files:/var/lib/pgadmin \
  -e "[email protected]" -e "PGADMIN_DEFAULT_PASSWORD=secretpassword" \
  dpage/pgadmin4 

docker update --restart always pgadmin4

Using Docker Compose

Installing Docker Compose

Compose is not part of Docker engine, it's installed separately.

Side note, on Bellman I installed Docker engine using the Docker repository for the latest stable version. Currently I have 18.09.02.

To install Docker Compose on Debian, see https://docs.docker.com/compose/install/#install-compose for example, 

curl -L https://github.com/docker/compose/releases/download/1.24.0-rc1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version
docker-compose version 1.24.0-rc1, build 0f3d4dda

Compose file

Okay now that I have tested Geoserver, PostGIS, PostgreSQL, and pgadmin separately I am ready to roll them together with one Docker Compose file. Here is my file "docker-compose.yml", so far. 

# Match this to your docker engine, I am at 18.09.02 in Debian 9
version: '3.7'

services:
  db:
    image: mdillion/postgis:11
    environment:
      PGDATA: /var/lib/postgresql/data
      POSTGRES_PASSWORD: example
      #POSTGRES_USER:
      #POSTGRES_DB:
      #POSTGRES_INITDB_ARGS:
      #POSTGRES_INITDB_XLOGDIR:
    ports:
    - "5432:5432"
    volumes:
    - "postgis_files:/var/lib/postgresql/data"
 
  # I will want this behind an nginx reverse proxy instead of exposing the port
  geoserver:
    image: geoceg/geoserver:latest
    ports:
    - "8888:8080"

  pgadmin:
    image: pgadmin
    ports:
    - ""

Fire it up

docker-compose up -d

For this to be accessible I have to punch through the firewall Aurora on ports 8080 and 5433.

Connect to it to test it psql -U postgres -h localhost

In this case I have mapped the standard postgres port to the same port on the host, I can get away with this because there is no instance of Postgres already running here. Then I can connect to it as if it were running locally with the psql command.

From the psql prompt, I can check out how the postgis set up went with 

\c geonode_imports
select postgis_full_version();
POSTGIS="2.3.1 r15264" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10.1, released 2013/08/26" LIBXML="2.9.1" LIBJSON="0.11.99" RASTER

This looks reasonably complete and reasonably up to date. (Hmph, GDAL is a bit out of date, version 2 is out.)

You can run an interactive shell and look around. 

docker exec -it postgis /bin/bash

Plumbing the network

Create a network to hook together the containers 

docker network create pg
docker run --name postgis --net=pg -p 5432 -d mdillon/postgis
docker run --name geoserver --net=pg -p 8888:8080 -d geoceg/geoserver
docker network inspect pg

The two containers can see each other now with the names 'postgis' and 'geoserver' resolving correctly. The 'inspect' command shows the two containers are connected to the pg network.

Dealing with passwords

I don't want passwords to be coded into my Docker images, so how do I get them out of there? I know I am supposed to put them into environment settings but then how do I load the environment?

Setting up GeoServer on Linux

The downloadable binary for Linux includes jetty; this precludes the need for Tomcat. Indeed this is great for testing on a desktop but for any server application, I find that geoserver/java will eat ram and crash after a few days.

So I install Tomcat and expand the settings to give it lots of space.

Tomcat 8

I used to install Oracle manually (direct tarball download) and then was forced to install Tomcat from source, too. Otherwise doing "apt-get install tomcat7" got me an apt packaged outdated java.

Now I first set up the Oracle repo for Java and install it first so that everything can load from apt.

Fix for Debian 8.5, to get the add-apt-repository command (which is an Ubuntu thing) 

cd /tmp
wget http://blog.anantshri.info/content/uploads/2010/09/add-apt-repository.sh.txt
mv add-apt-repository.sh.txt /usr/local/sbin
chmod 755 /usr/local/sbin/add-apt-repository

Now I can install Oracle version 8 JDK as an apt package. 

apt-get install python-software-properties
add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java8-installer

Now since I have Oracle java installed as a package, I can install the tomcat8 package and it won't try to install whatever open sourced java it ordinarily installs. It will be happy to use Oracle's version.

Startup of tomcat fails at this point because JAVA_HOME is not set. They still use the old /etc/init.d /etc/default startup files, so I put my custom settings at the end of /etc/default/tomcat8: 

JAVA_HOME="/usr/lib/jvm/java-8-oracle"
GEOCEG_HOME="/home/geoceg"
GEOSERVER_HOME="$GEOCEG_HOME/geoserver"

JAVA_OPTS="-Djava.awt.headless=true -Xmx768m -Xrs -XX:PerfDataSamplingInterval=500 \          
 -XX:MaxPermSize=256m -Dorg.geotools.referencing.forceXY=true \                                       
 -DGEOSERVER_DATA_DIR=$GEOCEG_HOME/geoserver/data \                                                   
 -DGEOWEBCACHE_DATA_DIR=$GEOCEG_HOME/geowebcache \                                                    
 -DGEOEXPLORER_DATA=$GEOCEG_HOME/geoexplorer \                                                        
 "

Then start it up, fix any problems, try try again 

systemctl start tomcat8

At this point there should be a servlet container running at http://dart.wildsong.biz:8080/

How big is my heap now?

java -XX:+PrintFlagsFinal -version 2>&1 | grep -i MaxHeapSize

Geoserver 2.9.1

Download stable release of the WAR file from http://geoserver.org/ -- Today that would be here: 

cd ~/Downloads
wget http://downloads.sourceforge.net/project/geoserver/GeoServer/2.9.1/geoserver-2.9.1-war.zip?r=http%3A%2F%2Fgeoserver.org%2Frelease%2F2.9.1%2F&ts=1473614756&use_mirror=superb-sea2
unzip geoserver*

Based on the settings I used in /etc/default/tomcat8 above I need to install the WAR file at /var/lib/tomcat8/webapps/ and the rest of the files go in /home/geoceg Create the folders first, when you move the WAR file it will be deployed automatically and start searching for the folders. 

mkdir /home/geoceg
chdir /home/geoceg
mkdir -p geoexplorer geowebcache geoserver/data
chown -R tomcat8.wildsong /home/geoceg/*

mv geoserver.war /var/lib/tomcat8/webapps
chown root.root /var/lib/tomcat8/webapps/geoserver.war

systemctl restart tomcat8

If deployment succeeds then some files will be created in /home/geoceg/geoserver for you, and you should be able to see an empty geoserver at http://dart.wildsong.biz:8080/geoserver/

As you work on getting everything correct, the log file /var/lib/tomcat8/logging/catalina.out will be most helpful.


Change admin password now.

You have to look up the one that was created for you when geoserver.war ran the first time. It is in geoserver/data/security/masterpw.info Login as root in the browser and change it now.

Next I have to set up my own workspace to serve my PostGIS data.

Connecting GeoServer to PostGIS

Some notes on using WFS-T http://blogs.law.harvard.edu/jreyes/2007/08/03/geotools-wfs-t-update-request/

GeoServer has

  • workspaces aka namespaces
  • stores aka datastores
  • layers

These are about organizing the data accessible to GeoServer.

You store data in stores and a store has to be in a workspace. A PostGIS store connects to a database, and the tables therein become accessible as layers (once you publish them)

So for example, you can create a workspace for Oregon and then put all your state-level data into stores that are in the Oregon workspace.

OpenStreetMap

I have already set up the OpenStreetMap data for western US in PostGIS. There is a nice page to get started here: http://blog.geoserver.org/2009/01/30/geoserver-and-openstreetmap/

You will need SLD files, this one is uselessly outdated github.com/mitchelljj/openstreetmap-sld.git Trying https://github.com/ROGUE-JCTD/geoserver-osm-styles.git

Here is what I have so far, http://dart.wildsong.biz:8080/geoserver/osm/wms?service=WMS&version=1.1.0&request=GetMap&layers=osm:planet_osm_roads&styles=&bbox=-1.4011076E7,3631139.0,-1.1346024E7,6468468.0&width=721&height=768&srs=EPSG:3857&format=application/openlayers

Other PostGIS data

To create a connection to data in your PostGIS server you have to know what's in there. You can poke around with psql if you are text kind of guy or you can use pgAdmin III, like so:

Retrieved from "https://wiki.wildsong.biz//index.php?title=GeoServer&oldid=11607"