GeoServer: Difference between revisions

From Wildsong
Jump to navigationJump to search
← Older edit
VisualWikitext
Brian Wilson (talk | contribs)
Bureaucrats, restrict, Administrators, uploaders
11,548 edits
mNo edit summary
Brian Wilson (talk | contribs)
Bureaucrats, restrict, Administrators, uploaders
11,548 edits
 
(89 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Overview ==
== Overview ==


GeoServer is the Java based GIS web server that I am using with [[PostgreSQL]] and [[PostGIS]] for the [[Mapping Vietnam]] project
GeoServer is what I used with [[PostgreSQL]] and [[PostGIS]] for the [[Mapping Vietnam]] project.


Home page: http://geoserver.org/
Home page: http://geoserver.org/
where they say
"GeoServer is a Java-based software server that allows users to view and '''edit geospatial data'''. Using open standards set forth by the Open Geospatial Consortium (OGC), GeoServer allows for great flexibility in map creation and data sharing."


2012 Sep -- 2.2 installed<br>
The crucial feature for me is that it allows edits via [[#WFS-T]]. That means I can hook up OpenLayers web clients to a database and allow them to create and edit data.
2011 Oct -- I am working with 2.1.2 now<br>
2009 Feb 06 -- I decided to take a crack at installing GeoServer 1.7.2 today.
 
"GeoServer is a Java-based software server that allows users to view and '''edit geospatial data'''. Using open standards set forth by the Open Geospatial Consortium (OGC), GeoServer allows for great flexibility in map creation and data sharing."


The crucial feature is that it allows edits via WFS-T. [[Mapserver]] does not.
== History ==


WFS-T comments: http://weblogs.java.net/blog/2006/01/09/geoserver-wfs-t
* 2019 Mar -- Migrated to Docker Compose.
* 2016 Sep -- updated Dart with 2.9.1 (latest stable release)
* 2012 Sep -- 2.2 installed on Kilchis and Dart.<br>
* 2011 Oct -- I am working with 2.1.2 now on HuPI<br>
* 2009 Feb 06 -- I decided to take a crack at installing GeoServer 1.7.2 today.


== Sample GeoServer site ==
== Example GeoServer site (not mine!) ==


The [http://ride.trimet.org/ Portland TriMet system map] is on GeoServer via the OpenGeo stack.
The [http://ride.trimet.org/ Portland TriMet system map] is on GeoServer via the [[OpenGeo]] stack.


This is from the Portland Trimet site...
This is from the Portland Trimet site...
GeoServer, OpenLayers, TileCache, PostgreSQL, PostGIS, GDAL, Extjs, Freemarker, Solr, Antlr, Balance  Ant, Http Client, Tomcat, Apache, and Cent OS are the underlying technologies used on this application. We're indebted to the contributors of these (and many other: eclipse, firefox  and firebug to name a few) open source projects for sharing their efforts and expertise.
GeoServer, [[OpenLayers]], TileCache, PostgreSQL, PostGIS, GDAL, Extjs, Freemarker, Solr, Antlr, Balance  Ant, Http Client, Tomcat, Apache, and CentOS are the underlying technologies used on this application. We're indebted to the contributors of these (and many other: eclipse, firefox  and firebug to name a few) open source projects for sharing their efforts and expertise.


== Geoserver Capabilities ==
== Some GeoServer Capabilities ==


* WCS = Web Coverage Service = queryable feature data (OGC)
* WCS = Web Coverage Service = queryable feature data (OGC)
Line 33: Line 35:
I think WCS can return features that are related from several feature classes whereas WFS returns features from one class only??
I think WCS can return features that are related from several feature classes whereas WFS returns features from one class only??


If you are implementing a tile-based client you should probably use WMTS or maybe TMS. WMS-C was a OSGEO draft.
=== Map vs feature services ===
 
What if you want features to overlay a map but want symbology defined by the server?
You define styles and then pick the correct encoding.
 
With '''WFS''' there are several GML flavours, GeoJSON, shapefiles and CSV. None of these work for me because they only encode the geography and not the visual properties. I probably want to use a '''WMS''' encoded with KML.
 
Lew's approach is to convert the KML into a custom JSON format on the server for transfer. There must be a better way than that. But it has to be compatible with the Google Maps Android API too.
 
The official way to put markers and shapes onto a Google map is via their API. I don't want to go the route of putting shapes onto the screen ignoring the API because I hate having to do all the projection and transformation stuff on my own.
 
Being able to directly read [[GPX]] files would be a good thing.
 
=== Tiles ===
 
If you are implementing a tile-based client you should probably use TMS or maybe WMTS. WMS-C was a OSGEO draft.
WMTS is more complex so use TMS unless you really need WMTS features.


On the server side for performance and the best cartography you'd probably be better off pre-rendering and caching all the tiles. What's the best way to do that? Tilemill or gdal2tiles ? What's the best way to serve the prerendered tiles? TileCache? Or just use a web server (Apache)?
On the server side for performance and the best cartography you'd probably be better off pre-rendering and caching all the tiles. What's the best way to do that? Tilemill or gdal2tiles ? What's the best way to serve the prerendered tiles? TileCache? Or just use a web server (Apache)?


See http://TileCache.org/
TMS reference implementation: see http://TileCache.org/
 
== Geoserver in Docker ==


== Status ==
The docker-compose set up includes geoserver, geowebcache, postgis, and pgadmin4.
See the full project on github, https://github.com/Wildsong/docker-geoserver


I have installed GeoServer 2.1.2 on HuPI.org, using the WAR file and Tomcat 7. It's running on the Oracle Java JRE. I had to upgrade my virtual machine account because 512M was not enough to run it. At work I have 2.2 running.
I had a reverse proxy built into the compose setup but realized I needed a proxy for the other services running on [[Bellman]]
so now I have a generic proxy running in its own container.
Because GeoServer is tucked behind a proxy, it is accessible "everywhere" (currently it's behind my [[Firewall]] settings but that's a different story).


I have gotten data loaded into PostGIS and published it in GeoServer.
=== PostGIS ===


Then I need to add the new layer into the map.
I keep more PostGIS notes here: [[PostGIS]].


The Apache server and raster tiles will remain at Hupi.
See https://hub.docker.com/r/mdillon/postgis
I will probably stop using this container, because I want the bleeding edge in my development server. (I live for pain)
I can probably even go to PostGIS 3.0 now!!


== Setting up GeoServer ==
From Postgis.org today: ''Although this release will work for PostgreSQL 9.4 and above, to take full advantage of what PostGIS 2.5 offers, you should be running PostgreSQL 11beta4+ and GEOS 3.7.0 which were released recently.
Best served with PostgreSQL 11 beta4 and pgRouting 2.6.1.''


I found instructions in the PostGIS book. Page 324-
In my current Docker, I can see using "docker inspect geoserver-db" that PostgreSQL is at 11.2 and from psql I can see GEOS is at 3.7.
As mentioned above I currently use the WAR file method.


=== Tomcat 7 ===
postgres=# '''SELECT postgis_full_version();'''
POSTGIS="2.5.1 r17027" [EXTENSION] PGSQL="110" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 2.1.2, released 2016/10/24" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" RASTER
(1 row)


I removed OpenJDK (which is purported to be suboptimal for use with GeoServer) and installed Oracle JDK 7. You only need JRE not JDK for Tomcat now, but I needed JDK for ESRI MDB support in GDAL.
See also its parent https://hub.docker.com/_/postgres/


Removing the OpenJDK also removed about 400 pounds of other packages, including Tomcat 6. This is okay really, since Tomcat 7 is the latest release anyway.
You can run the database independent of the compose configuration to test it.
These commands put the data and config files outside of the container, but still in a docker volume.


Download Tomcat 7 in binary form and unpack it where it will live.
docker pull mdillon/postgis:11
docker volume create postgis_files
docker run -d --name postgis -v postgis_files:/var/lib/postgresql/data \
  -e PGDATA=/var/lib/postgresql/data -e POSTGRES_PASSWORD='''secret''' mdillon/postgis:11


cd /tmp
Run psql to talk to the database;
wget http://mirror.uoregon.edu/apache/tomcat/tomcat-7/v7.0.23/bin/apache-tomcat-7.0.23.tar.gz
I generally have psql installed on the host and run it directly but you might not want to do that.
sudo mkdir /usr/local/java
cd /usr/local/java
sudo tar xzvf /tmp/apache-tomcat-7.0.23.tar.gz
ln -s apache-tomcat-7.0.23 tomcat7
cd tomcat7
Note -- fix the users and passwords up in conf/tomcat-users.xml
chown -R tomcat.tomcat temp work logs webapps
chgrp -R tomcat conf
chmod 750 conf
chmod g+w conf/*
su - tomcat
sh bin/startup.sh


Now try to connect to your server on port 8080, eg http://bellman:8080/
docker exec -it postgis /usr/bin/psql -U postgres
Do bin/shutdown.sh and finish configuration.
'''
How big is my heap?''' I can see on Hupi it's not enough for Geoserver!


java -XX:+PrintFlagsFinal -version 2>&1 | grep -i MaxHeapSize
Docker makes setting up PostGIS shockingly easy. You'll want pgadmin to maintain it.


This shows 256M, but in the tomcat startup file it's set to 128M.
==== ArcGIS and PostGIS ====


I create a /etc/default/tomcat file containing
After completing the PostGIS setup, I can now add PostGIS layers directly into ArcMap '''for read-only access'''.
# Brian cooked up this file from the original written for tomcat6           
I do this by creating a "Database Connection" in ArcCatalog; then I can see the layers in PostGIS and just drag them straight into ArcMap.  
I cannot '''EDIT''' them though, ArcMap does not support that directly. It might if you use the Data Interop extension but I don't have that.
TOMCAT_GROUP=tomcat
TOMCAT_USER=tomcat
# 2011-Nov-01 bwilson --Geoserver is hungry for more RAM!!!                 
JAVA_OPTS="-Djava.awt.headless=true -Xmx512m -XX:+UseConcMarkSweepGC"
JAVA_HOME="/usr/local/java/jre1.6.0_29"
JVM_TMP=/tmp/tomcat7-temp


I make a startup file in /etc/init.d based on an existing one.
If you have an ArcGIS Server license you can enable access to PostGIS in the PostgreSQL server. I can't afford the license fees.


<pre>
=== pgadmin ===
#!/bin/sh
#
# /etc/init.d/tomcat7 -- startup script for the Tomcat servlet engine
#
# Brian hacked this version to run Tomcat 7
#
# Written by Miquel van Smoorenburg <[email protected]>.
# Modified for Debian GNU/Linux by Ian Murdock <[email protected]>.
# Modified for Tomcat by Stefan Gybas <[email protected]>.
# Modified for Tomcat6 by Thierry Carrez <[email protected]>.
# Additional improvements by Jason Brittain <[email protected]>.
#
### BEGIN INIT INFO
# Provides:          tomcat7
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:    $local_fs $remote_fs $network
# Should-Start:      $named
# Should-Stop:      $named
# Default-Start:    2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start Tomcat.
# Description:      Start the Tomcat servlet engine.
### END INIT INFO


set -e
Until version 4, pgadmin was a desktop app. Now it's a javascript app that can run still from desktop but I have it installed as a '''docker''' on [[Bellman]].


PATH=/bin:/usr/bin:/sbin:/usr/sbin
I use this container: https://hub.docker.com/r/dpage/pgadmin4
NAME=tomcat
DESC="Tomcat servlet engine"
DEFAULT=/etc/default/$NAME
JVM_TMP=/tmp/tomcat-temp


if [ `id -u` -ne 0 ]; then
=== WFS-T ===
echo "You need root privileges to run this script"
exit 1
fi
# Make sure tomcat is started with system locale
if [ -r /etc/default/locale ]; then
. /etc/default/locale
export LANG
fi


. /lib/lsb/init-functions
Goal: Set up an example to allow storing dots from ol-react to a [[PostGIS]] instance and to an [[ArcGIS Enterprise instance]].
I don't care if they are WFS-T services or not as long as my web map works correctly. For PostGIS it looks easiest to use GeoServer and WFS-T.
That's what set me on this whole docker-compose path; I can now set up the server infrastructure in 10 minutes.


if [ -r /etc/default/rcS ]; then
# QGIS
. /etc/default/rcS
* Create an empty "web_markers" point layer in temp memory in QGIS, then use the QGIS database tool to export it to PostGIS.
fi
* Remove the PostGIS based layer from QGIS.
# GeoServer
* Create a Workspace. Enable WFS on the workspace.
* Create a Store. Connect it to PostGIS using the gis_owner role and password, so that it has full read/write access on PostGIS.
* Make sure WFS is enabled and Service Level is set to Transactional or Complete, (Services->WFS)
* Publish sure web_markers in Layers.
* Created a role "PUBLIC_ACCESS" and give it READ and WRITE access to the web_markers layer.
* Create a "markers" user with password and assign the PUBLIC_ACCESS role to it.
# In a browser,
* test with this URL: https://maps.wildsong.biz/geoserver/wfs?service=wfs&version=1.2.0&request=GetCapabilities which gives results for every layer
* and https://maps.wildsong.biz/geoserver/wfs?service=wfs&request=DescribeFeatureType&version=1.1.0&typeName=clatsop-wfs:web_markers with results only for that layer


In the GetCapabilities request there should be an operation name = "Transaction" section when transactions are enabled.


# The following variables can be overwritten in $DEFAULT
[[File:geoserver_data_security.png]]


# Run Tomcat as this user ID and group ID
In '''QGIS'''
TOMCAT_USER=tomcat
# I did "Layer->Add Layer->Add WFS Layer"
TOMCAT_GROUP=tomcat
# For "connection details" I used the URL https://maps.wildsong.biz/geoserver/wfs and left the authentication section alone.
# then I just selected the web_markers layer and clicked "Add" and "Closed".


# The first existing directory is used for JAVA_HOME (if JAVA_HOME is not
The parts of OpenLayers 3 that we need to support feature editing through WFS-T are:
# defined in $DEFAULT or in the environment)
#JDK_DIRS="/usr/lib/jvm/java-6-openjdk /usr/lib/jvm/java-6-sun /usr/lib/jvm/java-1.5.0-sun /usr/lib/j2sdk1.5-sun /usr/lib/j2sdk1.5-ibm"


# Look for the right JVM to use
ol.interaction.Draw
#for jdir in $JDK_DIRS; do
ol.interaction.Modify
#    if [ -r "$jdir/bin/java" -a -z "${JAVA_HOME}" ]; then
ol.format.WFS
# JAVA_HOME="$jdir"
#    fi
#done


# Directory where the Tomcat binary distribution resides
==== References that I found last night: ====
CATALINA_HOME=/usr/local/$NAME


# Directory for per-instance configuration files and webapps
Geoserver docs https://docs.geoserver.org/latest/en/user/services/wfs/reference.html
# This would be in /var/lib/tomcat6 if this were a Debian package
CATALINA_BASE=/usr/local/$NAME


# Use the Java security manager? (yes/no)
An article on Medium https://medium.com/@goldrydigital/wfs-t-with-openlayers-3-16-6fb6a820ac58
TOMCAT_SECURITY=no


# Default Java options
Discussion re an example https://github.com/openlayers/openlayers/issues/8854
# Set java.awt.headless=true if JAVA_OPTS is not set so the
# Xalan XSL transformer can work without X11 display on JDK 1.4+
# It also looks like the default heap size of 64M is not enough for most cases
# so the maximum heap size is set to 128M
if [ -z "$JAVA_OPTS" ]; then
JAVA_OPTS="-Djava.awt.headless=true -Xmx128M"
fi


# End of variables that can be overwritten in $DEFAULT
Boundless https://boundlessgeo.com/2014/06/openlayers-editing-wfs-t/
and http://suite.opengeo.org/opengeo-docs/sysadmin/wfst/index.html


# overwrite settings from default file
Stack Exchange https://gis.stackexchange.com/questions/141031/getting-transaction-support-enabled-in-wfs-on-geoserver-boundless-opengeosuite
if [ -f "$DEFAULT" ]; then
. "$DEFAULT"
fi
export JAVA_HOME


if [ ! -f "$CATALINA_HOME/bin/bootstrap.jar" ]; then
=== Serving MVT Vector Tiles ===
log_failure_msg "$NAME is not installed"
exit 1
fi


POLICY_CACHE="$CATALINA_BASE/work/catalina.policy"
MVT = Mapbox Vector Tiles


if [ -z "$CATALINA_TMPDIR" ]; then
You need to add a plugin to GeoServer for this to work.
CATALINA_TMPDIR="$JVM_TMP"
I had to create a new set up for GeoServer so that I could add plugins to the Docker image.
fi


# Set the JSP compiler if set in the tomcat6.default file
https://docs.geoserver.org/latest/en/user/extensions/vectortiles/install.html
if [ -n "$JSP_COMPILER" ]; then
JAVA_OPTS="$JAVA_OPTS -Dbuild.compiler=\"$JSP_COMPILER\""
fi


SECURITY=""
Once it's installed you can go look under Tile Layers at layers you have published
if [ "$TOMCAT_SECURITY" = "yes" ]; then
and pull up the "Tile Caching" tab. Then you should see Tile Image Formats as a list
SECURITY="-security"
that now includes "application/vnd.mapbox-vector-tile", turn that one one (and any
fi
others you wish to use of course).


# Define other required variables
If you do a preview of the layer you can select the PBF entries to test it and to find the correct URL to use in OpenLayers (or whereever...).
CATALINA_PID="/var/run/$NAME.pid"
CATALINA_SH="$CATALINA_HOME/bin/catalina.sh"


# Look for Java Secure Sockets Extension (JSSE) JARs
My (currently firewalled) test of Clatsop county is here: https://geoserver.wildsong.biz/geoserver/gwc/demo/clatsop_wm:taxlots?gridSet=EPSG:900913&format=application/vnd.mapbox-vector-tile
if [ -z "${JSSE_HOME}" -a -r "${JAVA_HOME}/jre/lib/jsse.jar" ]; then
    JSSE_HOME="${JAVA_HOME}/jre/"
fi


catalina_sh() {
Internally it looks like it's using this: https://geoserver.wildsong.biz/geoserver/gwc/service/wmts?REQUEST=GetTile&SERVICE=WMTS&VERSION=1.0.0&LAYER=clatsop_wm:taxlots&STYLE=&TILEMATRIX=EPSG:900913:10&TILEMATRIXSET=EPSG:900913&FORMAT=application/vnd.mapbox-vector-tile&TILECOL=161&TILEROW=366
# Escape any double quotes in the value of JAVA_OPTS
JAVA_OPTS="$(echo $JAVA_OPTS | sed 's/\"/\\\"/g')"


AUTHBIND_COMMAND=""
What's WMTS? https://en.wikipedia.org/wiki/Web_Map_Tile_Service
if [ "$AUTHBIND" = "yes" -a "$1" = "start" ]; then
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
AUTHBIND_COMMAND="/usr/bin/authbind --deep /bin/bash -c "
fi


# Define the command to run Tomcat's catalina.sh as a daemon
== Using Docker Compose ==
# set -a tells sh to export assigned variables to spawned shells.
TOMCAT_SH="set -a; JAVA_HOME=\"$JAVA_HOME\"; source \"$DEFAULT\"; \
CATALINA_HOME=\"$CATALINA_HOME\"; \
CATALINA_BASE=\"$CATALINA_BASE\"; \
JAVA_OPTS=\"$JAVA_OPTS\"; \
CATALINA_PID=\"$CATALINA_PID\"; \
CATALINA_TMPDIR=\"$CATALINA_TMPDIR\"; \
LANG=\"$LANG\"; JSSE_HOME=\"$JSSE_HOME\"; \
cd \"$CATALINA_BASE\"; \
\"$CATALINA_SH\" $@"


if [ "$AUTHBIND" = "yes" -a "$1" = "start" ]; then
Using Compose is AWESOME, it allows me to group services together and hook them up.
TOMCAT_SH="'$TOMCAT_SH'"
I have successfully started and stopped the whole thing several times now. It tears down
fi
(basically throws everything away) your containers when you say "docker-compose down"
but since I have persisted the data in volumes, everything comes right back online with
"docker-compose up"! Did I say "awesome" already?


# Run the catalina.sh script as a daemon
=== Installing Docker Compose ===
set +e
touch "$CATALINA_PID" "$CATALINA_BASE"/logs/catalina.out
chown $TOMCAT_USER "$CATALINA_PID" "$CATALINA_BASE"/logs/catalina.out
start-stop-daemon --start -b -u "$TOMCAT_USER" -g "$TOMCAT_GROUP" \
-c "$TOMCAT_USER" -d "$CATALINA_TMPDIR" -p "$CATALINA_PID" \
-x /bin/bash -- -c "$AUTHBIND_COMMAND $TOMCAT_SH"
status="$?"
set +a -e
return $status
}


case "$1" in
'''Compose''' is not part of '''Docker engine''', it's installed separately.
  start)
if [ -z "$JAVA_HOME" ]; then
log_failure_msg "no JDK found - please set JAVA_HOME"
exit 1
fi


if [ ! -d "$CATALINA_BASE/conf" ]; then
''Side note, on Bellman I installed Docker engine using the Docker repository for the ''latest stable'' version.'' Currently I have 18.09.02.
log_failure_msg "invalid CATALINA_BASE: $CATALINA_BASE"
exit 1
fi


log_daemon_msg "Starting $DESC" "$NAME"
To install Docker Compose on Debian, see https://docs.docker.com/compose/install/#install-compose
if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
for example,
--user $TOMCAT_USER --exec "$JAVA_HOME/bin/java" \
curl -L https://github.com/docker/compose/releases/download/1.24.0-rc1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
>/dev/null; then
chmod +x /usr/local/bin/docker-compose
# Regenerate POLICY_CACHE file
  docker-compose --version
umask 022
'''docker-compose version 1.24.0-rc1, build 0f3d4dda'''
echo "// AUTO-GENERATED FILE from $CATALINA_BASE/conf/policy.d/" > "$POLICY_CACHE"
echo "" >> "$POLICY_CACHE"
cat $CATALINA_BASE/conf/policy.d/*.policy \
>> "$POLICY_CACHE"


# Remove / recreate JVM_TMP directory
=== Configure everything ===
rm -rf "$JVM_TMP"
mkdir -p "$JVM_TMP" || {
log_failure_msg "could not create JVM temporary directory"
exit 1
}
chown $TOMCAT_USER "$JVM_TMP"


catalina_sh start $SECURITY
My compose file includes all services to run GeoServer, including GeoServer itself, [[GeoWebCache]], PostGIS/PostgreSQL, PGAdmin4, and nginx.
sleep 5
        if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null; then
if [ -f "$CATALINA_PID" ]; then
rm -f "$CATALINA_PID"
fi
log_end_msg 1
else
log_end_msg 0
fi
else
        log_progress_msg "(already running)"
log_end_msg 0
fi
;;
  stop)
log_daemon_msg "Stopping $DESC" "$NAME"


set +e
Refer to its github repo at https://github.com/Geo-CEG/geoserver-compose for more information.
if [ -f "$CATALINA_PID" ]; then
start-stop-daemon --stop --pidfile "$CATALINA_PID" \
--user "$TOMCAT_USER" \
--retry=TERM/20/KILL/5 >/dev/null
if [ $? -eq 1 ]; then
log_progress_msg "$DESC is not running but pid file exists, cleaning up"
elif [ $? -eq 3 ]; then
PID="`cat $CATALINA_PID`"
log_failure_msg "Failed to stop $NAME (pid $PID)"
exit 1
fi
rm -f "$CATALINA_PID"
rm -rf "$JVM_TMP"
else
log_progress_msg "(not running)"
fi
log_end_msg 0
set -e
;;
  status)
set +e
start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null 2>&1
if [ "$?" = "0" ]; then


if [ -f "$CATALINA_PID" ]; then
git clone https://github.com/Geo-CEG/geoserver-compose.git geoserver
    log_success_msg "$DESC is not running, but pid file exists."
cd geoserver
exit 1
# Read the README.md file.
else
# Copy files into the right places. dotenv.sample -> .env
    log_success_msg "$DESC is not running."
# Edit .env
exit 3
fi
else
log_success_msg "$DESC is running with pid `cat $CATALINA_PID`"
fi
set -e
        ;;
  restart|force-reload)
if [ -f "$CATALINA_PID" ]; then
$0 stop
sleep 1
fi
$0 start
;;
  try-restart)
        if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null; then
$0 start
fi
        ;;
  *)
log_success_msg "Usage: $0 {start|stop|restart|try-restart|force-reload|status}"
exit 1
;;
esac


exit 0
==== Fire it up ====
</pre>


docker-compose up -d


Once you have Tomcat running you can move on to getting Geoserver set up.
I have mapped the standard postgres port to the same port on the host,
based on the assumption there is no instance of PostgreSQL already running on the host.
I can connect to PostgreSQL as if it were running locally with the psql command; like this: psql -U postgres -h localhost


=== Geoserver 2.2 ===
From the psql prompt, I can check out how the postgis set up went with


Setting up Geoserver is very easy, download the files and put them in the webapps directory. Use the Tomcat manager page to start it.
select postgis_full_version();
POSTGIS="2.5.1 r17027" [EXTENSION] PGSQL="110" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016"
GDAL="GDAL 2.1.2, released 2016/10/24" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" TOPOLOGY RASTER
(1 row)


cd /usr/local/java/tomcat/webapps
This looks good.
wget http://downloads.sourceforge.net/geoserver/geoserver-2.2-war.zip
I can test the [https://maps.wildsong.biz/ nginx home page] and geoserver and geowebcache and pgadmin4 at this point and they all work, too.
unzip geoserver-2.2-war.zip


Once it is there, Tomcat will notice it and start it up. Once started you can access it here
You can also use "docker inspect" for example


  http://bellman:8080/geoserver/
  docker inspect geoserver-db | grep VERSION
      "GOSU_VERSION=1.11",
      "PG_VERSION=11.2-1.pgdg90+1",
      "POSTGIS_VERSION=2.5.1+dfsg-1.pgdg90+1"


'''Change admin password now.'''
=== Dealing with passwords ===


You have to look up the one that was created for you when geoserver.war ran the first time. It is in geoserver/data/security/masterpw.info Login as root in the browser and change it now.
I keep the initial passwords (for example, for Postgres user) in the .env file; again, see the README.md


This is new in 2.2, in 2.1 there was a default password and you had to edit a file to change it.
== Accessing GeoServer WFS from ArcGIS Pro ==


Next I have to set up my own workspace to serve my PostGIS data.
There is probably an expensive extension (Interop?) for ArcMap to use WFS but I don't have it.
AGS appears to have support; I am trying it out.


== Need to fix ==
As is traditional for Esri I had to loosen security settings on GeoServer. Now anyone can see the services.
Write access is still restricted.


/opt/apache/tomcat/webapps/geoserver/data/gwc-layers
I followed instructions to [https://pro.arcgis.com/en/pro-app/help/data/services/add-wfs-services.htm#GUID-FC7E9995-1105-4E11-85E3-910C81F8517B create a WFS server connection] then I [https://pro.arcgis.com/en/pro-app/help/data/services/use-wfs-services.htm#ESRI_SECTION2_0167C63B86F9484D9619953D6A2D8723 viewed WFS layers in the Contents Pane]. See View->Catalog Pane->Servers


INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
== Setting up GeoServer on Linux ==


== Connecting GeoServer to PostGIS ==
I used to have some instructions in this page to help me set up GeoServer directly in Linux but now that I know how to use Docker, I deleted them. If you really feel the need to do it the hard way, use View History up there in the tabs and check for the version older than Feb 19 2019.
 
== Getting data into GeoServer ==
 
=== Adding raster files ===
 
https://0-proquest.safaribooksonline.com.marinet.lib.ca.us/book/gis/9781788297370/adding-your-data/a96658aa_e2d4_4f15_a2c6_9edfa3a47e81_xhtml?query=((geoserver)+AND+(raster))#snippet
 
[https://www.e-education.psu.edu/geog585/node/710 Walkthrough: Creating tiles with GeoServer] at Penn State.
 
I want to do this so I can test WPS geoprocessing, not because I think it's the best way to distribute raster data, okay?
 
# I downloaded some data from DOGAMI months ago, a portion of LiDAR for Astoria. So I have a georeferenced TIFF file.
# Copy the .tif and .tfw files up onto GeoServer, to the Docker volume; on my server /home/docker/volumes/geoserver_data/_data/clatsop/
# Data (in left bar) -> Stores -> Add new store
# Raster Data Sources -> GeoTIFF
# I chose the clatsop workspace
# I used "Browse" to create the URL: file:clatsop/irving.tif
# Save
# This takes me to a New Layer page showing the layer name "irving" and I click on "Publish".
 
I accepted the rest of the defaults that GeoServer filled in for me.
 
I did a preview of the layer and it came up blank. Then I found the new layer in "Tile Layers" under "Tile Caching" and told it to build tiles for zoom levels 4-19 and waited for it to complete (5 minutes or so). ''NOW''... I still get a blank preview map! "Scale 1:NaN" is wrong. '''WRONG place'''!
 
To preview, Tile Caching -> Tile Layers -> select the layer -> Select a preview mode (PNG for example) then a previewer pops up.
 
[[Image:Geoserver_tile_preview.png]]
 
== Using GeoServer with PostGIS ==


Some notes on using WFS-T http://blogs.law.harvard.edu/jreyes/2007/08/03/geotools-wfs-t-update-request/
Some notes on using WFS-T http://blogs.law.harvard.edu/jreyes/2007/08/03/geotools-wfs-t-update-request/
Line 419: Line 302:
A PostGIS store connects to a database, and the tables therein become accessible as layers (once you publish them)
A PostGIS store connects to a database, and the tables therein become accessible as layers (once you publish them)


So you can create a workspace for Oregon and then put all your state-level data into stores that are in the Oregon workspace.
So for example, you can create a workspace for Oregon and then put all your state-level data into stores that are in the Oregon workspace.
 
== Queries in GeoServer ==
 
=== CQL ===
 
You can add CQL filters to a REST URL to filter on attributes in a WFS service, you can use spatial operators too.


To create a connection to data in your PostGIS server you have to know what's in there. You can poke around with psql if you are text kind of guy or you can use pgAdmin III, like so:
http://geoserver.wildsong.biz/geoserver/wfs?service=WFS&version=1.0.0&request=GetFeature&typeName=clatsop_wm:county_boundary&maxFeatures=50&outputFormat=application/json&CQL_FILTER=INTERSECTS(geom,collectGeometries(queryCollection('clatsop_wm:parks','geom','IN(''parks.1'')')))


[[Image:pgAdmin_III.png|thumb]]


See the docs [https://docs.geoserver.org/stable/en/user/filter/ecql_reference.html#filter-ecql-reference]


https://github.com/mappingmashups/simple-things-on-maps


[[Category: GIS]]
[[Category:GIS]]
[[Category:OpenLayers]]

Latest revision as of 21:56, 4 December 2019

Overview

GeoServer is what I used with PostgreSQL and PostGIS for the Mapping Vietnam project.

Home page: http://geoserver.org/ where they say "GeoServer is a Java-based software server that allows users to view and edit geospatial data. Using open standards set forth by the Open Geospatial Consortium (OGC), GeoServer allows for great flexibility in map creation and data sharing."

The crucial feature for me is that it allows edits via #WFS-T. That means I can hook up OpenLayers web clients to a database and allow them to create and edit data.

History

  • 2019 Mar -- Migrated to Docker Compose.
  • 2016 Sep -- updated Dart with 2.9.1 (latest stable release)
  • 2012 Sep -- 2.2 installed on Kilchis and Dart.
  • 2011 Oct -- I am working with 2.1.2 now on HuPI
  • 2009 Feb 06 -- I decided to take a crack at installing GeoServer 1.7.2 today.

Example GeoServer site (not mine!)

The Portland TriMet system map is on GeoServer via the OpenGeo stack.

This is from the Portland Trimet site... GeoServer, OpenLayers, TileCache, PostgreSQL, PostGIS, GDAL, Extjs, Freemarker, Solr, Antlr, Balance Ant, Http Client, Tomcat, Apache, and CentOS are the underlying technologies used on this application. We're indebted to the contributors of these (and many other: eclipse, firefox and firebug to name a few) open source projects for sharing their efforts and expertise.

Some GeoServer Capabilities

  • WCS = Web Coverage Service = queryable feature data (OGC)
  • WFS = Web Feature Service = vector data (OGC)
  • WMS = Web Map Service = delivers rasters (not tiles) (OGC)
  • TMS = Tile map service = delivers raster tiles (OSGEO)
  • WMS-C = Web Map Service - Cached = WMS with tile caching (OSGEO draft)
  • WMTS = Web Map Tile service (OGC)

I think WCS can return features that are related from several feature classes whereas WFS returns features from one class only??

Map vs feature services

What if you want features to overlay a map but want symbology defined by the server? You define styles and then pick the correct encoding.

With WFS there are several GML flavours, GeoJSON, shapefiles and CSV. None of these work for me because they only encode the geography and not the visual properties. I probably want to use a WMS encoded with KML.

Lew's approach is to convert the KML into a custom JSON format on the server for transfer. There must be a better way than that. But it has to be compatible with the Google Maps Android API too.

The official way to put markers and shapes onto a Google map is via their API. I don't want to go the route of putting shapes onto the screen ignoring the API because I hate having to do all the projection and transformation stuff on my own.

Being able to directly read GPX files would be a good thing.

Tiles

If you are implementing a tile-based client you should probably use TMS or maybe WMTS. WMS-C was a OSGEO draft. WMTS is more complex so use TMS unless you really need WMTS features.

On the server side for performance and the best cartography you'd probably be better off pre-rendering and caching all the tiles. What's the best way to do that? Tilemill or gdal2tiles ? What's the best way to serve the prerendered tiles? TileCache? Or just use a web server (Apache)?

TMS reference implementation: see http://TileCache.org/

Geoserver in Docker

The docker-compose set up includes geoserver, geowebcache, postgis, and pgadmin4. See the full project on github, https://github.com/Wildsong/docker-geoserver

I had a reverse proxy built into the compose setup but realized I needed a proxy for the other services running on Bellman so now I have a generic proxy running in its own container. Because GeoServer is tucked behind a proxy, it is accessible "everywhere" (currently it's behind my Firewall settings but that's a different story).

PostGIS

I keep more PostGIS notes here: PostGIS.

See https://hub.docker.com/r/mdillon/postgis I will probably stop using this container, because I want the bleeding edge in my development server. (I live for pain) I can probably even go to PostGIS 3.0 now!!

From Postgis.org today: Although this release will work for PostgreSQL 9.4 and above, to take full advantage of what PostGIS 2.5 offers, you should be running PostgreSQL 11beta4+ and GEOS 3.7.0 which were released recently. Best served with PostgreSQL 11 beta4 and pgRouting 2.6.1.

In my current Docker, I can see using "docker inspect geoserver-db" that PostgreSQL is at 11.2 and from psql I can see GEOS is at 3.7. 

postgres=# SELECT postgis_full_version();
POSTGIS="2.5.1 r17027" [EXTENSION] PGSQL="110" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 2.1.2, released 2016/10/24" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" RASTER
(1 row)

See also its parent https://hub.docker.com/_/postgres/

You can run the database independent of the compose configuration to test it. These commands put the data and config files outside of the container, but still in a docker volume. 

docker pull mdillon/postgis:11
docker volume create postgis_files
docker run -d --name postgis -v postgis_files:/var/lib/postgresql/data \
  -e PGDATA=/var/lib/postgresql/data -e POSTGRES_PASSWORD=secret mdillon/postgis:11

Run psql to talk to the database; I generally have psql installed on the host and run it directly but you might not want to do that. 

docker exec -it postgis /usr/bin/psql -U postgres

Docker makes setting up PostGIS shockingly easy. You'll want pgadmin to maintain it.

ArcGIS and PostGIS

After completing the PostGIS setup, I can now add PostGIS layers directly into ArcMap for read-only access. I do this by creating a "Database Connection" in ArcCatalog; then I can see the layers in PostGIS and just drag them straight into ArcMap. I cannot EDIT them though, ArcMap does not support that directly. It might if you use the Data Interop extension but I don't have that.

If you have an ArcGIS Server license you can enable access to PostGIS in the PostgreSQL server. I can't afford the license fees.

pgadmin

Until version 4, pgadmin was a desktop app. Now it's a javascript app that can run still from desktop but I have it installed as a docker on Bellman.

I use this container: https://hub.docker.com/r/dpage/pgadmin4

WFS-T

Goal: Set up an example to allow storing dots from ol-react to a PostGIS instance and to an ArcGIS Enterprise instance. I don't care if they are WFS-T services or not as long as my web map works correctly. For PostGIS it looks easiest to use GeoServer and WFS-T. That's what set me on this whole docker-compose path; I can now set up the server infrastructure in 10 minutes.

  1. QGIS
  • Create an empty "web_markers" point layer in temp memory in QGIS, then use the QGIS database tool to export it to PostGIS.
  • Remove the PostGIS based layer from QGIS.
  1. GeoServer
  • Create a Workspace. Enable WFS on the workspace.
  • Create a Store. Connect it to PostGIS using the gis_owner role and password, so that it has full read/write access on PostGIS.
  • Make sure WFS is enabled and Service Level is set to Transactional or Complete, (Services->WFS)
  • Publish sure web_markers in Layers.
  • Created a role "PUBLIC_ACCESS" and give it READ and WRITE access to the web_markers layer.
  • Create a "markers" user with password and assign the PUBLIC_ACCESS role to it.
  1. In a browser,

In the GetCapabilities request there should be an operation name = "Transaction" section when transactions are enabled.

In QGIS

  1. I did "Layer->Add Layer->Add WFS Layer"
  2. For "connection details" I used the URL https://maps.wildsong.biz/geoserver/wfs and left the authentication section alone.
  3. then I just selected the web_markers layer and clicked "Add" and "Closed".

The parts of OpenLayers 3 that we need to support feature editing through WFS-T are:

ol.interaction.Draw ol.interaction.Modify ol.format.WFS

References that I found last night:

Geoserver docs https://docs.geoserver.org/latest/en/user/services/wfs/reference.html

An article on Medium https://medium.com/@goldrydigital/wfs-t-with-openlayers-3-16-6fb6a820ac58

Discussion re an example https://github.com/openlayers/openlayers/issues/8854

Boundless https://boundlessgeo.com/2014/06/openlayers-editing-wfs-t/ and http://suite.opengeo.org/opengeo-docs/sysadmin/wfst/index.html

Stack Exchange https://gis.stackexchange.com/questions/141031/getting-transaction-support-enabled-in-wfs-on-geoserver-boundless-opengeosuite

Serving MVT Vector Tiles

MVT = Mapbox Vector Tiles

You need to add a plugin to GeoServer for this to work. I had to create a new set up for GeoServer so that I could add plugins to the Docker image.

https://docs.geoserver.org/latest/en/user/extensions/vectortiles/install.html

Once it's installed you can go look under Tile Layers at layers you have published and pull up the "Tile Caching" tab. Then you should see Tile Image Formats as a list that now includes "application/vnd.mapbox-vector-tile", turn that one one (and any others you wish to use of course).

If you do a preview of the layer you can select the PBF entries to test it and to find the correct URL to use in OpenLayers (or whereever...).

My (currently firewalled) test of Clatsop county is here: https://geoserver.wildsong.biz/geoserver/gwc/demo/clatsop_wm:taxlots?gridSet=EPSG:900913&format=application/vnd.mapbox-vector-tile

Internally it looks like it's using this: https://geoserver.wildsong.biz/geoserver/gwc/service/wmts?REQUEST=GetTile&SERVICE=WMTS&VERSION=1.0.0&LAYER=clatsop_wm:taxlots&STYLE=&TILEMATRIX=EPSG:900913:10&TILEMATRIXSET=EPSG:900913&FORMAT=application/vnd.mapbox-vector-tile&TILECOL=161&TILEROW=366

What's WMTS? https://en.wikipedia.org/wiki/Web_Map_Tile_Service

Using Docker Compose

Using Compose is AWESOME, it allows me to group services together and hook them up. I have successfully started and stopped the whole thing several times now. It tears down (basically throws everything away) your containers when you say "docker-compose down" but since I have persisted the data in volumes, everything comes right back online with "docker-compose up"! Did I say "awesome" already?

Installing Docker Compose

Compose is not part of Docker engine, it's installed separately.

Side note, on Bellman I installed Docker engine using the Docker repository for the latest stable version. Currently I have 18.09.02.

To install Docker Compose on Debian, see https://docs.docker.com/compose/install/#install-compose for example, 

curl -L https://github.com/docker/compose/releases/download/1.24.0-rc1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version
docker-compose version 1.24.0-rc1, build 0f3d4dda

Configure everything

My compose file includes all services to run GeoServer, including GeoServer itself, GeoWebCache, PostGIS/PostgreSQL, PGAdmin4, and nginx.

Refer to its github repo at https://github.com/Geo-CEG/geoserver-compose for more information. 

git clone https://github.com/Geo-CEG/geoserver-compose.git geoserver
cd geoserver
# Read the README.md file.
# Copy files into the right places. dotenv.sample -> .env
# Edit .env

Fire it up

docker-compose up -d

I have mapped the standard postgres port to the same port on the host, based on the assumption there is no instance of PostgreSQL already running on the host. I can connect to PostgreSQL as if it were running locally with the psql command; like this: psql -U postgres -h localhost

From the psql prompt, I can check out how the postgis set up went with 

select postgis_full_version();
POSTGIS="2.5.1 r17027" [EXTENSION] PGSQL="110" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016"
GDAL="GDAL 2.1.2, released 2016/10/24" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" TOPOLOGY RASTER
(1 row)

This looks good. I can test the nginx home page and geoserver and geowebcache and pgadmin4 at this point and they all work, too.

You can also use "docker inspect" for example 

docker inspect geoserver-db | grep VERSION
     "GOSU_VERSION=1.11",
     "PG_VERSION=11.2-1.pgdg90+1",
     "POSTGIS_VERSION=2.5.1+dfsg-1.pgdg90+1"

Dealing with passwords

I keep the initial passwords (for example, for Postgres user) in the .env file; again, see the README.md

Accessing GeoServer WFS from ArcGIS Pro

There is probably an expensive extension (Interop?) for ArcMap to use WFS but I don't have it. AGS appears to have support; I am trying it out.

As is traditional for Esri I had to loosen security settings on GeoServer. Now anyone can see the services. Write access is still restricted.

I followed instructions to create a WFS server connection then I viewed WFS layers in the Contents Pane. See View->Catalog Pane->Servers

Setting up GeoServer on Linux

I used to have some instructions in this page to help me set up GeoServer directly in Linux but now that I know how to use Docker, I deleted them. If you really feel the need to do it the hard way, use View History up there in the tabs and check for the version older than Feb 19 2019.

Getting data into GeoServer

Adding raster files

https://0-proquest.safaribooksonline.com.marinet.lib.ca.us/book/gis/9781788297370/adding-your-data/a96658aa_e2d4_4f15_a2c6_9edfa3a47e81_xhtml?query=((geoserver)+AND+(raster))#snippet

Walkthrough: Creating tiles with GeoServer at Penn State.

I want to do this so I can test WPS geoprocessing, not because I think it's the best way to distribute raster data, okay?

  1. I downloaded some data from DOGAMI months ago, a portion of LiDAR for Astoria. So I have a georeferenced TIFF file.
  2. Copy the .tif and .tfw files up onto GeoServer, to the Docker volume; on my server /home/docker/volumes/geoserver_data/_data/clatsop/
  3. Data (in left bar) -> Stores -> Add new store
  4. Raster Data Sources -> GeoTIFF
  5. I chose the clatsop workspace
  6. I used "Browse" to create the URL: file:clatsop/irving.tif
  7. Save
  8. This takes me to a New Layer page showing the layer name "irving" and I click on "Publish".

I accepted the rest of the defaults that GeoServer filled in for me.

I did a preview of the layer and it came up blank. Then I found the new layer in "Tile Layers" under "Tile Caching" and told it to build tiles for zoom levels 4-19 and waited for it to complete (5 minutes or so). NOW... I still get a blank preview map! "Scale 1:NaN" is wrong. WRONG place!

To preview, Tile Caching -> Tile Layers -> select the layer -> Select a preview mode (PNG for example) then a previewer pops up.

Using GeoServer with PostGIS

Some notes on using WFS-T http://blogs.law.harvard.edu/jreyes/2007/08/03/geotools-wfs-t-update-request/

GeoServer has

  • workspaces aka namespaces
  • stores aka datastores
  • layers

These are about organizing the data accessible to GeoServer.

You store data in stores and a store has to be in a workspace. A PostGIS store connects to a database, and the tables therein become accessible as layers (once you publish them)

So for example, you can create a workspace for Oregon and then put all your state-level data into stores that are in the Oregon workspace.

Queries in GeoServer

CQL

You can add CQL filters to a REST URL to filter on attributes in a WFS service, you can use spatial operators too.

http://geoserver.wildsong.biz/geoserver/wfs?service=WFS&version=1.0.0&request=GetFeature&typeName=clatsop_wm:county_boundary&maxFeatures=50&outputFormat=application/json&CQL_FILTER=INTERSECTS(geom,collectGeometries(queryCollection('clatsop_wm:parks','geom','IN(parks.1)')))


See the docs [1]

https://github.com/mappingmashups/simple-things-on-maps

Retrieved from "https://wiki.wildsong.biz//index.php?title=GeoServer&oldid=12137"