Geoprocessing: Difference between revisions

From Wildsong
Jump to navigationJump to search
Brian Wilson (talk | contribs)
Brian Wilson (talk | contribs)
 
(24 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[Category:ESRI]]
In this case "Geoprocessing" means doing specialized spatial processing on data as required in my web maps work.


Online help: http://webhelp.esri.com/arcgisdesktop/9.1/index.cfm
This page used to cover geoprocessing in the ESRI world, and it was mostly about tips for Python+ArcPy.
I rewrote it to document WPS (Web Processing Service) first in [[GeoServer]] and later on in ArcGIS Enterprise.


Different ways to do geoprocessing in ArcGIS
I have separate pages on building and using GDAL and PostGIS, see the GIS category.


# Basic stuff the buffering in ArcMap
I built a Docker Compose [[GeoServer]] for testing, and I have added plugins to it for WPS.  
# From ArcToolbox in either ArcMap or ArcCatalog
I also have been developing an OpenLayers Javascript client.
# Command line, either a shell or from a command window in ArcCatalog; if you run it in an ArcCatalog window it helps you type the right args. Not quite sure why anyone would want to use the command line this way...
# [[Model Builder]] can be used to invoke standard tools or to call your scripts.
# Scripting usually with [[Python]], then calling scripts from one of the above methods.


== TIPS ==
== Books ==


=== ArcCatalog Tools->Options->Geoprocessing ===
''Expert GeoServer'' by Ben Mearns, is a very short book (134 pages) covering primarily WPS and Tile Caching (two unrelated topics, both interesting).


Check the box to allow overwriting of files, so that you don't end up creating numbered files like water_1 and allstreets_3
== Penn State class ==


Set your default tools directory to something reasonable.
I am walking through this full detailed lesson plan: [https://www.e-education.psu.edu/geog585/ Open Web Mapping]
So far, very good!


=== ESRI ===
== Boundless tutorial ==


See also [[Model Builder]]
'''IGNORE''' should you run across this  [https://suite.opengeo.org/docs/latest/processing/contour/setup.html Contour Map setup] from Boundless Geospatial. In the Boundless docs it's difficult for a beginnto tell what is a tool they provide (such as Composer) and what things are actually part of GeoServer. Theoretically it's all open source but in practice what you find on Github tends to be nonfunctional.


27-Jun-2011 At ArcGIS release 10 they have a thing called ArcPy. Have to look at that now.
* Composer supports creating maps using GeoServer, it has a style editor
* WPS Builder is like Esri Model Builder


31-Dec-2007 update: ArcGIS Desktop 9.3 includes a new python wrapper module that makes working with ArcObjects a lot easier. I am using ActiveState 2.4.3 with it with good results. It breaks if you try to move up to 2.5.1.
== Browser-based geoprocessing ==


You now just use the python line "import arcgisscripting" instead of directly invoking the COM dispatch thing.
A couple options here are [https://github.com/bjornharrtell/jsts#readme jsts] and [http://turfjs.org/ turf]. (and Turf wraps JSTS)


====Old notes from ArcGIS 9.1 ====
JSTS is a Javascript port of the Java Topology Suite (JTS).


This guy seems to have slogged through a lot of development with it and collected useful notes: http://www.ollivier.co.nz/support/python.shtm
The Turf documentation wins though. And it's used in GeoMoose 3. (JSTS is a dependency so you get both if you use Turf.)


I can add a new one to his list. I was getting error messages on setting gp.workspace = 'foo' where ''foo'' is a valid path. It's because the gp object was hanging around in pythonwin's memory. You should be able to say 'del gp' but I gave up and restarted pythonwin to fix it.
=== Turf ===


When working with ArcToolbox 9.1 and Python 2.1 I have been having lots of crashes in PythonWin. Using it on a computer with ActiveState 2.4.3 I have not been experiencing the same problems. I am going to try installing 2.4x without removing 2.1 so I can fall back on it... I followed the instructions above.
Go to [http://turfjs.org/getting-started Getting Started]


It still crashes. Oh joy. It does not even crash faster.
'''NB All coordinates in Turf have to be in WGS84.''' This is a minor pain.


== Open source ==
Also the default units in "buffer" is '''"kilometers"''' not "meters", what a surprise! :-)


ESRI has broken down and started to use http://www.gdal.org OGR/GDAL] for some things. I need to get the Python bindings installed...
Sample (using Parcel packager)


'''23-Aug-2006''' Python+OGR probably not worth messing with right now.
import buffer from '@turf/buffer'
From the OGR site: ''The Python API isn't really well documented at this time, but parallels the C/C++ APIs. The interface classes can be browsed in the pymod/ogr.py (simple features) and pymod/osr.py (coordinate systems) python modules. The pymod/samples/assemblepoly.py sample script is one demonstration of using the python API.''
import { toWgs84, toMercator } from '@turf/projection'
import jsonWM from './assets/mygeojson.json' // this data is in EPSG:3857, Web Mercator
let json84 = toWgs84(jsonWM);
let buffered = buffer(json84, 10, {"units":"meters"});
let bufferedWM = toMercator(buffered);


=== ArcGIS 10 + Python + GDAL ===
https://blog.webkid.io/how-to-handle-geospatial-data-with-nodejs-and-turf/


==== Not using OSGeo4W ====
== Server-based geoprocessing ==


I am looking at using OSGeo4W for this. See http://trac.osgeo.org/osgeo4w/
=== CQL queries ===


I am installing their version of Python and the GDAL bindings for gdal 1.7
You can create CQL filters and pass them in as part of the REST queries you send to GeoServer.
They have 1.8 but no python bindings for it. Typical for Windows stuff. Always incomplete.  
You can also create new layers with CQL filters applied, for example you could create a new taxlots_astoria
layer from taxlots with a CQL query of "CITY=Astoria", or in your browser app you could add "cql_filter=CITY=Astoria" to the URL.


OH GRAND. OSGeo4W is built on Python 2.5.
=== WPS servers ===
I don't understand why Windows has not withered away and died. :-)


==== Trying the Pypi ====
WPS Servers include GeoServer.
Others are
http://zoo-project.org/ and
https://pywps.org/
Both of these are OSGEO projects, too.


First install the installer. Get the Windows version for Python 2.6 and run it.
GeoServer WPS comes with a slew of predefined JTS tools.
See http://pypi.python.org/pypi/setuptools


Add "C:\Python26\ArcGIS10.0;C:\Python26\ArcGIS10.0\Scripts" to your PATH. That allows your command shell to find and run easy_install. Make sure it's ahead of OSGeo4W so that it does not find Python2.5 instead.
I found a good tutorial here: https://geoserver.geo-solutions.it/edu/en/wps/wps.html


You have to have gdal 1.7 installed before you install Python gdal.
List available services on my server:
Turns out that's part of the OSGeo4W installation done in the previous section, so now all I have to do is figure out how to reference those files.
https://geoserver.wildsong.biz/geoserver/ows?service=wps&version=1.0.0&request=GetCapabilities


You have to manually edit the config file!
Describe the buffer service, in detail but also alas in XML (hard for a human to read):
Here is mine:
https://geoserver.wildsong.biz/geoserver/ows?service=wps&version=1.0.0&request=DescribeProcess&Identifier=JTS:buffer


[egg_info]
Code a request in request.xml and send it.
  tag_build =
  curl -H "Content-type: xml" -X POST -d@request.xml https://geoserver.wildsong.biz/geoserver/wps -o response.json
tag_date = 0
tag_svn_revision = 0
[build_ext]
gdal_config = ../../apps/gdal-config
''' include_dirs = C:/OSGeo4W/include'''
''' library_dirs = C:/OSGeo4W/lib'''


I added the last two lines.
Now I have a JSON file and I can drag it onto my "example3" map to see it.


Next step WOULD be to go install the Python packages using the installer.
== Use case: Find my neighbors ==
The easy_install GDAL" fails because it can't find the cpl_ports.h file that's installed with gdal in C:\OSGeo4W\


See http://pypi.python.org/pypi/GDAL/
# User selects a taxlot in the Javscript client.  
# This causes adjacent taxlots to be selected.
# Information about all selected taxlots appears in a table.


=== FWTools ===
First step, let's find the taxlot id of all our recently saved building footprints, I will pick one for the taxlot account number.


[http://fwtools.maptools.org/ FWtools] installs a GUI called OpenEV_FW and a shell environment. It comes with python 2.3.4 which is old. I think Frank does this because it's stable and test with his tools. Going to a newer version means the DLL won't work so you'd need to compile your own.
1. Create a view that selects new buildings.<br />
2. Select all the taxlots that intersect those buildings and list their taxlot id numbers.
CREATE OR REPLACE VIEW clatsop_wm.buildings_new AS SELECT * FROM clatsop_wm.buildings WHERE checked=3;
SELECT tl.tax_id, tl.account_id FROM clatsop_wm.taxlots AS tl, clatsop_wm.buildings_new AS bldg WHERE (
ST_Intersects(tl.geom, bldg.geom)
) ORDER BY tl.tax_id;
 
Okay so let's pick one taxlot with account_id 25127, and find others near it.
Using a distance of 30 meters picks up the taxlots across the street too.
 
SELECT tl.taxlot, tl.account_id FROM clatsop_wm.taxlots AS tl WHERE
    ST_DWithin(
      (SELECT geom FROM clatsop_wm.taxlots WHERE account_id=25127),
      tl.geom,
      30
    );
 
[[File:neighbors.png]]
 
 
[[Category:GIS]]

Latest revision as of 19:11, 20 September 2019

In this case "Geoprocessing" means doing specialized spatial processing on data as required in my web maps work.

This page used to cover geoprocessing in the ESRI world, and it was mostly about tips for Python+ArcPy. I rewrote it to document WPS (Web Processing Service) first in GeoServer and later on in ArcGIS Enterprise.

I have separate pages on building and using GDAL and PostGIS, see the GIS category.

I built a Docker Compose GeoServer for testing, and I have added plugins to it for WPS. I also have been developing an OpenLayers Javascript client.

Books

Expert GeoServer by Ben Mearns, is a very short book (134 pages) covering primarily WPS and Tile Caching (two unrelated topics, both interesting).

Penn State class

I am walking through this full detailed lesson plan: Open Web Mapping So far, very good!

Boundless tutorial

IGNORE should you run across this Contour Map setup from Boundless Geospatial. In the Boundless docs it's difficult for a beginnto tell what is a tool they provide (such as Composer) and what things are actually part of GeoServer. Theoretically it's all open source but in practice what you find on Github tends to be nonfunctional.

  • Composer supports creating maps using GeoServer, it has a style editor
  • WPS Builder is like Esri Model Builder

Browser-based geoprocessing

A couple options here are jsts and turf. (and Turf wraps JSTS)

JSTS is a Javascript port of the Java Topology Suite (JTS).

The Turf documentation wins though. And it's used in GeoMoose 3. (JSTS is a dependency so you get both if you use Turf.)

Turf

Go to Getting Started

NB All coordinates in Turf have to be in WGS84. This is a minor pain.

Also the default units in "buffer" is "kilometers" not "meters", what a surprise! :-)

Sample (using Parcel packager)

import buffer from '@turf/buffer'
import { toWgs84, toMercator } from '@turf/projection'
import jsonWM from './assets/mygeojson.json' // this data is in EPSG:3857, Web Mercator
let json84 = toWgs84(jsonWM);
let buffered = buffer(json84, 10, {"units":"meters"});
let bufferedWM = toMercator(buffered);

https://blog.webkid.io/how-to-handle-geospatial-data-with-nodejs-and-turf/

Server-based geoprocessing

CQL queries

You can create CQL filters and pass them in as part of the REST queries you send to GeoServer. You can also create new layers with CQL filters applied, for example you could create a new taxlots_astoria layer from taxlots with a CQL query of "CITY=Astoria", or in your browser app you could add "cql_filter=CITY=Astoria" to the URL.

WPS servers

WPS Servers include GeoServer. Others are http://zoo-project.org/ and https://pywps.org/ Both of these are OSGEO projects, too.

GeoServer WPS comes with a slew of predefined JTS tools.

I found a good tutorial here: https://geoserver.geo-solutions.it/edu/en/wps/wps.html

List available services on my server: https://geoserver.wildsong.biz/geoserver/ows?service=wps&version=1.0.0&request=GetCapabilities

Describe the buffer service, in detail but also alas in XML (hard for a human to read): https://geoserver.wildsong.biz/geoserver/ows?service=wps&version=1.0.0&request=DescribeProcess&Identifier=JTS:buffer

Code a request in request.xml and send it.

curl -H "Content-type: xml" -X POST [email protected] https://geoserver.wildsong.biz/geoserver/wps -o response.json

Now I have a JSON file and I can drag it onto my "example3" map to see it.

Use case: Find my neighbors

  1. User selects a taxlot in the Javscript client.
  2. This causes adjacent taxlots to be selected.
  3. Information about all selected taxlots appears in a table.

First step, let's find the taxlot id of all our recently saved building footprints, I will pick one for the taxlot account number.

1. Create a view that selects new buildings.
2. Select all the taxlots that intersect those buildings and list their taxlot id numbers.

CREATE OR REPLACE VIEW clatsop_wm.buildings_new AS SELECT * FROM clatsop_wm.buildings WHERE checked=3;
SELECT tl.tax_id, tl.account_id FROM clatsop_wm.taxlots AS tl, clatsop_wm.buildings_new AS bldg WHERE (
		ST_Intersects(tl.geom, bldg.geom)
) ORDER BY tl.tax_id;

Okay so let's pick one taxlot with account_id 25127, and find others near it. Using a distance of 30 meters picks up the taxlots across the street too.

SELECT tl.taxlot, tl.account_id FROM clatsop_wm.taxlots AS tl WHERE 
   ST_DWithin(
      (SELECT geom FROM clatsop_wm.taxlots WHERE account_id=25127),
      tl.geom,
      30
   );

File:Neighbors.png