GeoServer

From Wildsong
Jump to navigationJump to search

Overview

GeoServer is what I used with PostgreSQL and PostGIS for the Mapping Vietnam project.

Home page: http://geoserver.org/ where they say "GeoServer is a Java-based software server that allows users to view and edit geospatial data. Using open standards set forth by the Open Geospatial Consortium (OGC), GeoServer allows for great flexibility in map creation and data sharing."

The crucial feature for me is that it allows edits via #WFS-T. That means I can hook up OpenLayers web clients to a database and allow them to create and edit data.

History

  • 2019 Feb -- Finishing the migration to Docker Compose.
  • 2017 Sep -- Setting up Docker Compose version
  • 2016 Sep -- updated Dart with 2.9.1 (latest stable release)
  • 2012 Sep -- 2.2 installed on Kilchis and Dart.
  • 2011 Oct -- I am working with 2.1.2 now on HuPI
  • 2009 Feb 06 -- I decided to take a crack at installing GeoServer 1.7.2 today.

Example GeoServer site (not mine!)

The Portland TriMet system map is on GeoServer via the OpenGeo stack.

This is from the Portland Trimet site... GeoServer, OpenLayers, TileCache, PostgreSQL, PostGIS, GDAL, Extjs, Freemarker, Solr, Antlr, Balance Ant, Http Client, Tomcat, Apache, and CentOS are the underlying technologies used on this application. We're indebted to the contributors of these (and many other: eclipse, firefox and firebug to name a few) open source projects for sharing their efforts and expertise.

Some GeoServer Capabilities

  • WCS = Web Coverage Service = queryable feature data (OGC)
  • WFS = Web Feature Service = vector data (OGC)
  • WMS = Web Map Service = delivers rasters (not tiles) (OGC)
  • TMS = Tile map service = delivers raster tiles (OSGEO)
  • WMS-C = Web Map Service - Cached = WMS with tile caching (OSGEO draft)
  • WMTS = Web Map Tile service (OGC)

I think WCS can return features that are related from several feature classes whereas WFS returns features from one class only??

Map vs feature services

What if you want features to overlay a map but want symbology defined by the server? You define styles and then pick the correct encoding.

With WFS there are several GML flavours, GeoJSON, shapefiles and CSV. None of these work for me because they only encode the geography and not the visual properties. I probably want to use a WMS encoded with KML.

Lew's approach is to convert the KML into a custom JSON format on the server for transfer. There must be a better way than that. But it has to be compatible with the Google Maps Android API too.

The official way to put markers and shapes onto a Google map is via their API. I don't want to go the route of putting shapes onto the screen ignoring the API because I hate having to do all the projection and transformation stuff on my own.

Being able to directly read GPX files would be a good thing.

Tiles

If you are implementing a tile-based client you should probably use TMS or maybe WMTS. WMS-C was a OSGEO draft. WMTS is more complex so use TMS unless you really need WMTS features.

On the server side for performance and the best cartography you'd probably be better off pre-rendering and caching all the tiles. What's the best way to do that? Tilemill or gdal2tiles ? What's the best way to serve the prerendered tiles? TileCache? Or just use a web server (Apache)?

TMS reference implementation: see http://TileCache.org/

Building the Geo-CEG Geoserver Docker images

There will be 4 Docker containers in the finished project, pulled together with Compose.

  1. mdillon/postgis@latest
  2. geoceg/postgres will contain PostgreSQL with the ESRI SDE additions to make it compatible with ArcGIS
  3. geoceg/geoserver which contains geoserver and geowebcache
  4. dpage/pgadmin4

Outside of Docker I will use Debian nginx to proxy these services and hide behind its TLS encryption with proper certificates for https://maps.wildsong.biz and https://bellman.wildsong.biz

Geoserver

I created a docker image and put it on github, https://github.com/Geo-CEG/docker-geoserver and it autobuilds an image at https://hub.docker.com/r/geoceg/geoserver/ and I continue to refine and update that image as needed in this project.

Test it independently with 

docker run -t -p 8888:8080 --name=geoserver -v geoserver_files:/geoserver geoceg/geoserver:latest

I proxy port 8888 through nginx on Bellman so that it shows up as https://maps.wildsong.biz/geoserver/web/

Because it's tucked behind this proxy, it is accessible everywhere (based on my Firewall settings but that's a different story). This is the relevant block of code in nginx conf, I have similar blocks for geowebcache and pgadmin4 

# Act as a proxy for a Docker instance of GeoServer
   location /geoserver/ {
       proxy_pass http://maps.wildsong.biz:8888/geoserver/;
       proxy_redirect      off;
       proxy_set_header    Host                    $host;
       proxy_set_header    X-Real-IP               $remote_addr;
       proxy_set_header    X-Forwarded-For         $proxy_add_x_forwarded_for;
       proxy_set_header    X-Forwarded-Proto       $scheme;
       proxy_set_header    X-Script-Name           /pgadmin4;
   }

Upgrade

I just updated the Github project from 2.13.2 to 2.14.2 and want to upgrade my installed image. 

docker stop geoserver
docker rm geoserver
docker pull geoceg/geoserver:latest

Then issue the same Docker run (or compose) used before.

PostGIS

See https://hub.docker.com/r/mdillon/postgis

See also its parent https://hub.docker.com/_/postgres/

These commands put the data and config files outside of the container. 

docker pull mdillon/postgis:11
docker volume create postgis_files

docker run -d --name postgis -v postgis_files:/var/lib/postgresql/data \
  -e PGDATA=/var/lib/postgresql/data -e POSTGRES_PASSWORD=secret mdillon/postgis:11

Run psql to talk to the database; 

docker exec -it postgis /usr/bin/psql -U postgres

This command actually starts a second container and links to the first one to do the same thing - give you psql. 

docker run -it --link postgis:postgres --rm postgres \
   sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres -W'

Either way once you get a prompt, you can do the quick test to see what's installed, 

postgres=# SELECT postgis_full_version();

POSTGIS="2.5.1 r17027" [EXTENSION] PGSQL="110" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016"

GDAL="GDAL 2.1.2, released 2016/10/24" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" TOPOLOGY RASTER (1 row)

Docker makes setting up PostGIS shockingly easy. You'll want pgadmin to maintain it.

ArcGIS SDE in PostgreSQL

After completing the previous section (PostGIS), I can now add PostGIS layers directly into ArcMap. I do this by creating a "Database Connection" in ArcCatalog; then I can see the layers in PostGIS and just drag them straight into ArcMap. I cannot EDIT them though, ArcMap does not support that directly. It might if you use the Data Interop extension but I don't have that so it matters naught to me.

I should be able to spatially enable PostGIS for ArcGIS though, referring to:

Requirements: https://desktop.arcgis.com/en/system-requirements/latest/database-requirements-postgresql.htm It wants Ubuntu 16.04.3, PostgreSQL 10.3 and PostGIS 2.4

That's correct, it now REQUIRES PostGIS. This seems wrong to me in the natural order of things, they are using both PostgreSQL and PostGIS.

This page tells me to install a DLL which is stupid since I am running on Linux https://desktop.arcgis.com/en/arcmap/latest/manage-data/gdbs-in-postgresql/get-started-gdb-in-postgresql.htm

I think I need this https://desktop.arcgis.com/en/arcmap/latest/manage-data/databases/add-the-st-geometry-type-to-a-postgresql-database.htm

I am going to try using the same image (different version) that I used for GeoServer, it's not built on Ubuntu but that might not matter. I going to build my own Dockerfile either way; so go look at this github project: Geo-CEG/postgis-arcsde.

pgadmin

Until version 4, pgadmin was a desktop app. Now it's a javascript app that can run still from desktop but I have it installed as a docker on Bellman.

I use this container: https://hub.docker.com/r/dpage/pgadmin4 

docker pull dpage/pgadmin4
docker volume create pgadmin_files

docker run -d -p 8889:80 --name=pgadmin4 -v pgadmin_files:/var/lib/pgadmin \
  -e "[email protected]" -e "PGADMIN_DEFAULT_PASSWORD=secretpassword" \
  dpage/pgadmin4

I hide the Docker behind nginx on Bellman, the service runs internally at http://maps.wildsong:8889/ but it shows up (for me) at https://maps.wildsong.biz/pgadmin4/

WFS-T

What I want to do is set up a minimal example storing dots from ol-react to a PostGIS instance and to an ArcGIS Enterprise instance. I don't care if they are WFS-T services or not as long as my web map works correctly. For PostGIS it looks easiest to use GeoServer and WFS-T. That's what set me on this whole docker-compose path; I can now set up the server infrastructure in 10 minutes.

  1. Over the weekend (not using my ArcGIS Desktop license) I created an empty "web_markers" point layer in temp memory in QGIS, then I used the QGIS database tool to export it to PostGIS.
  2. In GeoServer, I made sure WFS was enabled and Service Level is set to Transactional or Complete, (Services->WFS) and made sure web_markers was visible in layers.
  3. I did a test with this URL: https://maps.wildsong.biz/geoserver/wfs?service=wfs&version=1.2.0&request=GetCapabilities which gives me results for every layer
  4. and https://maps.wildsong.biz/geoserver/wfs?service=wfs&request=DescribeFeatureType&version=1.1.0&typeName=Clatsop:web_markers with results only for that layer

In the GetCapabilities request there should be an operation name = "Transaction" section when transactions are enabled.

There is also a security issue. I opened up write permission for ROLE_ANONYMOUS -- you probably don't want to do it that way, but now QGIS can write to WFS-T!!

In QGIS

  1. I did "Layer->Add Layer->Add WFS Layer"
  2. For "connection details" I used the URL https://maps.wildsong.biz/geoserver/wfs and left the authentication section alone.
  3. then I just selected the web_markers layer and clicked "Add" and "Closed".

The parts of OpenLayers 3 that we need to support feature editing through WFS-T are:

ol.interaction.Draw ol.interaction.Modify ol.format.WFS

References that I found last night:

Geoserver docs https://docs.geoserver.org/latest/en/user/services/wfs/reference.html

An article on Medium https://medium.com/@goldrydigital/wfs-t-with-openlayers-3-16-6fb6a820ac58

Discussion re an example https://github.com/openlayers/openlayers/issues/8854

Boundless https://boundlessgeo.com/2014/06/openlayers-editing-wfs-t/ and http://suite.opengeo.org/opengeo-docs/sysadmin/wfst/index.html

Stack Exchange https://gis.stackexchange.com/questions/141031/getting-transaction-support-enabled-in-wfs-on-geoserver-boundless-opengeosuite

Using Docker Compose

Installing Docker Compose

Compose is not part of Docker engine, it's installed separately.

Side note, on Bellman I installed Docker engine using the Docker repository for the latest stable version. Currently I have 18.09.02.

To install Docker Compose on Debian, see https://docs.docker.com/compose/install/#install-compose for example, 

curl -L https://github.com/docker/compose/releases/download/1.24.0-rc1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version
docker-compose version 1.24.0-rc1, build 0f3d4dda

Compose file

Now that I have tested Geoserver, PostGIS, PostgreSQL, and pgadmin separately I am ready to roll them together with one Docker Compose file. Here is my file "docker-compose.yml", so far. 

# Match this to your docker engine, I am at 18.09.02 in Debian 9
version: '3.7'

services:
  db:
    image: mdillion/postgis:11
    environment:
      PGDATA: /var/lib/postgresql/data
      POSTGRES_PASSWORD: ''your_secret_here''
      #POSTGRES_USER:
      #POSTGRES_DB:
      #POSTGRES_INITDB_ARGS:
      #POSTGRES_INITDB_XLOGDIR:
    ports:
    - "5432:5432"
    volumes:
    - "postgis_files:/var/lib/postgresql/data"
 
  # I will want this behind an nginx reverse proxy instead of exposing the port
  geoserver:
    image: geoceg/geoserver:latest
    ports:
    - "8888:8080"

  dpage/pgadmin4 

  pgadmin4:
    image: dpage/pgadmin4
    environment:
      PGADMIN_DEFAULT_EMAIL: ''your_login@somewhere''
      PGADMIN_DEFAULT_PASSWORD=''your_password''
    ports:
    - "8889:80"
    volumes:
    - pgadmin_files:/var/lib/pgadmin

Fire it up

docker-compose up -d

In this case I have mapped the standard postgres port to the same port on the host, There is no instance of Postgres already running here. Then I can connect to it as if it were running locally with the psql command.

From the psql prompt, I can check out how the postgis set up went with 

select postgis_full_version();
POSTGIS="2.5.1 r17027" [EXTENSION] PGSQL="110" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016"
GDAL="GDAL 2.1.2, released 2016/10/24" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" TOPOLOGY RASTER
(1 row)

This looks reasonably complete and reasonably up to date. I can test geoserver and geowebcache and pgadmin4 at this point and they all work, too.

You can also use "docker inspect" for example 

docker inspect postgis | grep VERSION
bwilson@bellman:~/source/docker/docker-geoserver$  docker inspect postgis | grep VERSION
                "GOSU_VERSION=1.11",
                "PG_VERSION=11.2-1.pgdg90+1",
                "POSTGIS_VERSION=2.5.1+dfsg-1.pgdg90+1"

Dealing with passwords

I don't want passwords to be coded into my Docker images, so how do I get them out of there? I know I am supposed to put them into environment settings but then how do I load the environment?

Setting up GeoServer on Linux

I used to have some instructions in this page to help me set up GeoServer directly in Linux but now that I know how to use Docker, I deleted them. If you really feel the need to do it the hard way, use View History up there in the tabs and check for the version older than Feb 19 2019.

Using GeoServer with PostGIS

Some notes on using WFS-T http://blogs.law.harvard.edu/jreyes/2007/08/03/geotools-wfs-t-update-request/

GeoServer has

  • workspaces aka namespaces
  • stores aka datastores
  • layers

These are about organizing the data accessible to GeoServer.

You store data in stores and a store has to be in a workspace. A PostGIS store connects to a database, and the tables therein become accessible as layers (once you publish them)

So for example, you can create a workspace for Oregon and then put all your state-level data into stores that are in the Oregon workspace.

Retrieved from "https://wiki.wildsong.biz//index.php?title=GeoServer&oldid=11626"