Geoprocessing: Difference between revisions

From Wildsong
Jump to navigationJump to search
Brian Wilson (talk | contribs)
mNo edit summary
Brian Wilson (talk | contribs)
 
(34 intermediate revisions by the same user not shown)
Line 1: Line 1:
Online help: http://webhelp.esri.com/arcgisdesktop/9.1/index.cfm
In this case "Geoprocessing" means doing specialized spatial processing on data as required in my web maps work.


The script so far:
This page used to cover geoprocessing in the ESRI world, and it was mostly about tips for Python+ArcPy.
I rewrote it to document WPS (Web Processing Service) first in [[GeoServer]] and later on in ArcGIS Enterprise.


<pre>
I have separate pages on building and using GDAL and PostGIS, see the GIS category.
# ---------------------------------------------------------------------------
# StructId.py
# Created on: Tue Feb 14 2006 05:06:58 PM
#  (generated by ArcGIS/ModelBuilder)
# ---------------------------------------------------------------------------


# Import system modules
I built a Docker Compose [[GeoServer]] for testing, and I have added plugins to it for WPS.  
import sys, string, os, win32com.client
I also have been developing an OpenLayers Javascript client.


# Create the Geoprocessor object
== Books ==
gp = win32com.client.Dispatch("esriGeoprocessing.GpDispatch.1")


#
''Expert GeoServer'' by Ben Mearns, is a very short book (134 pages) covering primarily WPS and Tile Caching (two unrelated topics, both interesting).
gp.workspace = "H:/BrianWilson/Workspace/Collections/Collections.mdb/Collections"
out_workspace = "C:/Temp/NewShapefiles"


# Load required toolboxes...
== Penn State class ==
#gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Data Management Tools.tbx")


# Process: Calculate Field...
I am walking through this full detailed lesson plan: [https://www.e-education.psu.edu/geog585/ Open Web Mapping]
# the expressions here can only be very simple.
So far, very good!
# You can use variables in [square brackets], you can use constants, and you can use
# simple operators like + - / * and (for strings) &
# manhole_shp = "C:\\TEMP\\NewShapefiles\\manhole.shp"
#gp.CalculateField_management(manhole_shp, "STRUCT_ID", "[STRUCT_TYP]&\"-\"& [MH_LABEL]")


# Export all the feature classes from our geodatabase to individual shapefiles
== Boundless tutorial ==
fcs=gp.ListFeatureClasses()
fcs.reset()
fc=fcs.Next()
while fc:
    output_shapefile = out_workspace + '/' + fc
    gp.AddMessage("Working on " + fc + "...")
    gp.FeatureClassToShapefile_conversion(fc, out_workspace)
    fc = fcs.next()


# Rename the new shapefiles to their correct GISMO names
'''IGNORE''' should you run across this  [https://suite.opengeo.org/docs/latest/processing/contour/setup.html Contour Map setup] from Boundless Geospatial. In the Boundless docs it's difficult for a beginnto tell what is a tool they provide (such as Composer) and what things are actually part of GeoServer. Theoretically it's all open source but in practice what you find on Github tends to be nonfunctional.


# Calculate the missing field values
* Composer supports creating maps using GeoServer, it has a style editor
* WPS Builder is like Esri Model Builder


# Delete the fields that don't belong in GISMO output
== Browser-based geoprocessing ==
</pre>
 
A couple options here are [https://github.com/bjornharrtell/jsts#readme jsts] and [http://turfjs.org/ turf]. (and Turf wraps JSTS)
 
JSTS is a Javascript port of the Java Topology Suite (JTS).
 
The Turf documentation wins though. And it's used in GeoMoose 3. (JSTS is a dependency so you get both if you use Turf.)
 
=== Turf ===
 
Go to [http://turfjs.org/getting-started Getting Started]
 
'''NB All coordinates in Turf have to be in WGS84.''' This is a minor pain.
 
Also the default units in "buffer" is '''"kilometers"''' not "meters", what a surprise! :-)
 
Sample (using Parcel packager)
 
import buffer from '@turf/buffer'
import { toWgs84, toMercator } from '@turf/projection'
import jsonWM from './assets/mygeojson.json' // this data is in EPSG:3857, Web Mercator
let json84 = toWgs84(jsonWM);
let buffered = buffer(json84, 10, {"units":"meters"});
let bufferedWM = toMercator(buffered);
 
https://blog.webkid.io/how-to-handle-geospatial-data-with-nodejs-and-turf/
 
== Server-based geoprocessing ==
 
=== CQL queries ===
 
You can create CQL filters and pass them in as part of the REST queries you send to GeoServer.
You can also create new layers with CQL filters applied, for example you could create a new taxlots_astoria
layer from taxlots with a CQL query of "CITY=Astoria", or in your browser app you could add "cql_filter=CITY=Astoria" to the URL.
 
=== WPS servers ===
 
WPS Servers include GeoServer.
Others are
http://zoo-project.org/ and
https://pywps.org/
Both of these are OSGEO projects, too.
 
GeoServer WPS comes with a slew of predefined JTS tools.
 
I found a good tutorial here: https://geoserver.geo-solutions.it/edu/en/wps/wps.html
 
List available services on my server:
https://geoserver.wildsong.biz/geoserver/ows?service=wps&version=1.0.0&request=GetCapabilities
 
Describe the buffer service, in detail but also alas in XML (hard for a human to read):
https://geoserver.wildsong.biz/geoserver/ows?service=wps&version=1.0.0&request=DescribeProcess&Identifier=JTS:buffer
 
Code a request in request.xml and send it.
curl -H "Content-type: xml" -X POST [email protected] https://geoserver.wildsong.biz/geoserver/wps -o response.json
 
Now I have a JSON file and I can drag it onto my "example3" map to see it.
 
== Use case: Find my neighbors ==
 
# User selects a taxlot in the Javscript client.
# This causes adjacent taxlots to be selected.
# Information about all selected taxlots appears in a table.
 
First step, let's find the taxlot id of all our recently saved building footprints, I will pick one for the taxlot account number.
 
1. Create a view that selects new buildings.<br />
2. Select all the taxlots that intersect those buildings and list their taxlot id numbers.
CREATE OR REPLACE VIEW clatsop_wm.buildings_new AS SELECT * FROM clatsop_wm.buildings WHERE checked=3;
SELECT tl.tax_id, tl.account_id FROM clatsop_wm.taxlots AS tl, clatsop_wm.buildings_new AS bldg WHERE (
ST_Intersects(tl.geom, bldg.geom)
) ORDER BY tl.tax_id;
 
Okay so let's pick one taxlot with account_id 25127, and find others near it.
Using a distance of 30 meters picks up the taxlots across the street too.
 
SELECT tl.taxlot, tl.account_id FROM clatsop_wm.taxlots AS tl WHERE
    ST_DWithin(
      (SELECT geom FROM clatsop_wm.taxlots WHERE account_id=25127),
      tl.geom,
      30
    );
 
[[File:neighbors.png]]
 
 
[[Category:GIS]]

Latest revision as of 19:11, 20 September 2019

In this case "Geoprocessing" means doing specialized spatial processing on data as required in my web maps work.

This page used to cover geoprocessing in the ESRI world, and it was mostly about tips for Python+ArcPy. I rewrote it to document WPS (Web Processing Service) first in GeoServer and later on in ArcGIS Enterprise.

I have separate pages on building and using GDAL and PostGIS, see the GIS category.

I built a Docker Compose GeoServer for testing, and I have added plugins to it for WPS. I also have been developing an OpenLayers Javascript client.

Books

Expert GeoServer by Ben Mearns, is a very short book (134 pages) covering primarily WPS and Tile Caching (two unrelated topics, both interesting).

Penn State class

I am walking through this full detailed lesson plan: Open Web Mapping So far, very good!

Boundless tutorial

IGNORE should you run across this Contour Map setup from Boundless Geospatial. In the Boundless docs it's difficult for a beginnto tell what is a tool they provide (such as Composer) and what things are actually part of GeoServer. Theoretically it's all open source but in practice what you find on Github tends to be nonfunctional.

  • Composer supports creating maps using GeoServer, it has a style editor
  • WPS Builder is like Esri Model Builder

Browser-based geoprocessing

A couple options here are jsts and turf. (and Turf wraps JSTS)

JSTS is a Javascript port of the Java Topology Suite (JTS).

The Turf documentation wins though. And it's used in GeoMoose 3. (JSTS is a dependency so you get both if you use Turf.)

Turf

Go to Getting Started

NB All coordinates in Turf have to be in WGS84. This is a minor pain.

Also the default units in "buffer" is "kilometers" not "meters", what a surprise! :-)

Sample (using Parcel packager)

import buffer from '@turf/buffer'
import { toWgs84, toMercator } from '@turf/projection'
import jsonWM from './assets/mygeojson.json' // this data is in EPSG:3857, Web Mercator
let json84 = toWgs84(jsonWM);
let buffered = buffer(json84, 10, {"units":"meters"});
let bufferedWM = toMercator(buffered);

https://blog.webkid.io/how-to-handle-geospatial-data-with-nodejs-and-turf/

Server-based geoprocessing

CQL queries

You can create CQL filters and pass them in as part of the REST queries you send to GeoServer. You can also create new layers with CQL filters applied, for example you could create a new taxlots_astoria layer from taxlots with a CQL query of "CITY=Astoria", or in your browser app you could add "cql_filter=CITY=Astoria" to the URL.

WPS servers

WPS Servers include GeoServer. Others are http://zoo-project.org/ and https://pywps.org/ Both of these are OSGEO projects, too.

GeoServer WPS comes with a slew of predefined JTS tools.

I found a good tutorial here: https://geoserver.geo-solutions.it/edu/en/wps/wps.html

List available services on my server: https://geoserver.wildsong.biz/geoserver/ows?service=wps&version=1.0.0&request=GetCapabilities

Describe the buffer service, in detail but also alas in XML (hard for a human to read): https://geoserver.wildsong.biz/geoserver/ows?service=wps&version=1.0.0&request=DescribeProcess&Identifier=JTS:buffer

Code a request in request.xml and send it.

curl -H "Content-type: xml" -X POST [email protected] https://geoserver.wildsong.biz/geoserver/wps -o response.json

Now I have a JSON file and I can drag it onto my "example3" map to see it.

Use case: Find my neighbors

  1. User selects a taxlot in the Javscript client.
  2. This causes adjacent taxlots to be selected.
  3. Information about all selected taxlots appears in a table.

First step, let's find the taxlot id of all our recently saved building footprints, I will pick one for the taxlot account number.

1. Create a view that selects new buildings.
2. Select all the taxlots that intersect those buildings and list their taxlot id numbers.

CREATE OR REPLACE VIEW clatsop_wm.buildings_new AS SELECT * FROM clatsop_wm.buildings WHERE checked=3;
SELECT tl.tax_id, tl.account_id FROM clatsop_wm.taxlots AS tl, clatsop_wm.buildings_new AS bldg WHERE (
		ST_Intersects(tl.geom, bldg.geom)
) ORDER BY tl.tax_id;

Okay so let's pick one taxlot with account_id 25127, and find others near it. Using a distance of 30 meters picks up the taxlots across the street too.

SELECT tl.taxlot, tl.account_id FROM clatsop_wm.taxlots AS tl WHERE 
   ST_DWithin(
      (SELECT geom FROM clatsop_wm.taxlots WHERE account_id=25127),
      tl.geom,
      30
   );

File:Neighbors.png