PostgreSQL

Which version?

Sometimes I compile my own installations of PostgreSQL, I have information on how and why to build from scratch in the PostGIS page.

Docker Migration

Using Docker is way better than installing software directly on the machine and makes working with advanced versions far simpler.

I am only using PostgreSQL on Bellman to host PostGIS and Geoserver right now, so follow the instructions here in this page: Geoserver to hook up Postgres + PostGIS + Geoserver + Geowebcache + pgadmin4.

I have notes on how to use Docker Compose to set up PostGIS + Geoserver and I need to migrate over to that setup soon and I need to add the pgadmin4 container described below.

I'm working with 9.6 now on my Debian installation. I am using the standard repo and install it from packages.

These are recommended: python-sqlalchemy postgresql-server-dev-all

First time

I usually have to immediately locate and modify pg_hba.conf and postgresql.conf to tune things. On Debian they are in /etc/postgresql/*/main/

See below to set a password on the postgres account.

Command line tools

psql gives you command access to PostgreSQL. Normally I set my system to trust local connections in the pg_hba.conf file then use psql -U postgres.

There are special psql commands starting with '\' so do '\?' to list them. Typically I use psql when I want to answer a few questions quickly. I also use it with pipes to load data. See the section on shp2psql in the PostGIS page.

Security

There are multiple levels (for me anyway)

1. Gateway firewall. My EdgeRouter forwards all inbound port 5432 to Bellman.
2. Bellman firewall. Bellman accepts all outside traffic from selected locations, in this case, my employer's IP address. start_firewall.sh (custom script)
3. PostgreSQL has to be set to listen on its network connection (not just localhost which is the default) in postgresql.conf
4. PostgreSQL filters are set in pg_hba.conf - in my case, I allow only specific IP addresses and my local network.

How to do stuff

Q. How can I debug queries and privileges?? HELP!!

On Debian try

tail -f /var/log/postgresql/postgresql-*.log

This will help you a LOT.

Q. What version do I have installed?

In psql,

SELECT version();
PostgreSQL 9.6.9 on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit

SELECT PostGIS_full_version();
POSTGIS="2.5.0dev r16556" [EXTENSION] PGSQL="96" GEOS="3.7.0dev-CAPI-1.11.0 b40bd637" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 2.2.4, released 2018/03/19" LIBXML="2.9.4" LIBJSON="0.12.1" RASTER

(1 row)

Q. How do I create a new user?

http://www.postgresql.org/docs/9.1/static/sql-createuser.html

psql -U postgres
CREATE USER bwilson WITH PASSWORD 'linenopise';
# and while we're in here...
GRANT ALL ON DATABASE somedatabase TO bwilson;

Q. How do I change postgres password?

sudo -s
su - postgres
psql -d template1
ALTER USER postgres WITH PASSWORD 'my_new_password';

Q. What databases are present on this server?

\l or \l+

Adding the' '+' to any command includes extra information, such as the amount of space used.

Q. Create a database and set its owner?

CLI

createdb -U postgres newdatabase

PSQL

psql -U postgres
CREATE DATABASE "newdatabase"
ALTER DATABASE "newdatabase" OWNER TO "new_owner"
GRANT ALL ON DATABASE "somedatabase" TO "new_owner";

Q. Make a user into a superuser?

psql -U postgres
ALTER USER "sde" WITH SUPERUSER;

Q. What tables are present in a database?

\d or \d+

Q. What columns are present in a table?

\dS TABLENAME

Q. How do I remove all the tables from a schema?

Drop the schema and then recreate it. drop schema x; create schema x; Then you also have to fix permissions on the schema.

DROP SCHEMA or_co_marion CASCADE;
CREATE SCHEMA or_co_marion
  AUTHORIZATION gis_owner;

GRANT ALL ON SCHEMA or_co_marion TO gis_owner;
GRANT USAGE ON SCHEMA or_co_marion TO gis_reader;

See also http://stackoverflow.com/questions/3327312/drop-all-tables-in-postgresql

Q. How do I do a query?

SQL works, so once I know a database and a table I can do regular SQL queries

\c crbc_spatial;
SELECT name FROM rivers WHERE name IS NOT NULL LIMIT 10;
          name           
--------------------------
Rock Creek
Long Meadow Creek
Weed Lake
Frenchman Creek
St. Martin Creek
South Fork Tenmile Creek
Sunflower Canyon
Chicken Springs Canyon
Cottonwood Canyon
Wood Canyon
(10 rows)

SELECT COUNT(*) FROM rivers WHERE name LIKE '% River'
 count 
-------
  312
(1 row)

More handy queries (Courtesy PostgreSQL 9 Admin. Cookbook, Packt Publishing)

SELECT current_database();
SELECT current_user;
SELECT version();

Upgrades

To move from 9.0.5 to 9.1.2, I did the dump-and-reload thing. I have only a little data that I need to preserve right now so this is easiest.

1. Get a list of my databases
2. Dump the ones with value to me with pg_dump
3. Stop the old server and start the new one
4. Load the data into the new server with pg_restore

I might choose to use pg_dumpall if I had lots of valuable data.

In my case I have about 20 databases but only needed to move over the one for trac. The rest (1 or 2) I can easily reload from the original shapefiles.

pg_dump -U postgres trac > trac.sql

Shutdown version 9.0 and set up version 9.1 using the instructions on the PostGIS page. Then load the databases back in

createdb -U postgres trac
psql -U postgres
CREATE USER trac WITH CREATEDB;
\password trac
\c trac
\i trac.sql

Associated tools and extensions

I have this old note, no idea if it is still relevant:

Navicat - a GUI front end for PostgreSQL. See http://www.navicat.com/ Includes report generation tools

pgadmin4

Used to be a desktop app, now it's a javascript app that can run from desktop but I have it installed as a docker on Bellman.

sudo mkdir /var/lib/vastra/pgadmin
sudo touch /var/lib/vastra/servers.json

docker pull dpage/pgadmin4
docker volume create pgadmin_files

docker run -d -p 8889:80 --name=pgadmin4 -v pgadmin_files:/var/lib/pgadmin \
  -v "/var/lib/vastra/pgadmin/servers.json:/servers.json" \
  -e "[email protected]" -e "PGADMIN_DEFAULT_PASSWORD=secretpassword" \
  dpage/pgadmin4 

docker update --restart always pgadmin4

Extensions

PostGIS - spatially enables PostgreSQL. (There is some spatial stuff built in.) PostGIS is one of many PostgreSQL Extensions

ArcGIS

It's 2019 and we're at 10.6 for these notes.

Esri uses PostgreSQL internally for its "Data Store".

Overall interop with PostgreSQL is good now. It works sort of like PostGIS, instead of adding POSTGIS to a database you have to enable ArcGIS on the database by adding the magic Esri "SDE" goo to it. If you don't and you just try to connect to a PostGIS database you will get ERROR: relation "sde.sde_layers" does not existin your PostgreSQL logs and see no tables in the database.

Setting up a connection

Make 100% sure you have username/password correct (duh!) and then select a database. If you don't get the creds in there then you won't be able to pick a database.

MS Access

I don't actually use Access and don't currently have a copy installed anywhere so someone should write up and add comments to this section.

article on using MS Access with PostgreSQL

The crucial aspect here is to install the PostgreSQL ODBC driver on Windows.

Things to note and perhaps deal with

1. Access users don't expect case-sensitivity.
2. There is no support for booleans in Access so you have to hack around this

Openoffice / Libreoffice Base

Use the SDBC driver. On Ubuntu this means selecting the package in Synaptic and installing it. libreoffice-sdbc-postgresql in my case. On Ubunu Base is not installed by default so I also installed libreoffice-base

The Datasource URL requested in Base should look like this:

host=myhostname.wildsong.biz port=5433 dbname=mydatabase

I have to know my username and password to test the connection. I have to tell it the name of a file to save the settings information in on the local computer.

Most critically at this point I have to know how to use both Base and PostgreSQL. :-)

Use case: Loading a spreadsheet into PostgreSQL

I have received a spreadsheet for the Mapping Vietnam project.

It has a GRID COORD column that must be converted to a point geometry, and it has a bunch of attribute columns.