View source for CC AGE

ArcGIS Enterprise consists of a set of services: Portal, ArcGIS Server, Web Adaptor, and Data Store.

''This is a copy of a page I keep at Clatsop, likely out of date.''

== History ==

* 2019-10-24 Upgraded to 10.7.1.
* 2018-02 Based on log files, I am thinking this is when we did the initial install at 10.6. It went into full production mode around May or so.

== Overview ==

There are basically four components. They can be deployed on separate servers but we have everything on one server, [[Cc-GIS]]
The components are
* Portal
* Server
* Data Store
* Web Adaptor

Web Adaptor installs inside of your web server (IIS here) and proxies the other services. You install 2, one for Portal and one for Server.
The other 3 components are separate applications running independently on the server. (One virtual server here. They can be installed on separate machines and you can run multiple copies of Server and Data Store to improve performance if you have lots of users.) We're also running SQL Server here and that could be moved into the county-wide SQL Server that IT maintains.

=== Portal ===

Portal is basically our own copy of ArcGIS Online running on our server. 
You use it to publish maps and data (feature classes, databases, files...). Also apps and services. 

(Under normal conditions) the URL for Portal: https://delta.co.clatsop.or.us/portal
If Web Adaptor is not running, you can access portal directly via https://delta.co.clatsop.or.us:7443/
but if web adaptor '''IS''' running then you just get redirected to the home page when you hit this URL.

==== Portal customizations ====

Note regarding bit.ly -- Portal still uses the API but bit.ly no longer supports API keys, so it won't work.

The config.js file on our server is here: c:/Program Files/ArcGIS/Portal/customizations/10.7.1/framework/webapps/arcgis#home/js/arcgisonline

To reread the config file you have to restart through the Windows control panel -> services

=== ArcGIS Server ===

ArcGIS Server is the core provider of all the services.

It generates maps and returns them via ArcGIS proprietary protocols and OGC protocols like WMS.
It exposes various data stores for use in ArcGIS Desktop and/or in Portal.

URL for ArcGIS Server Manager: https://delta.co.clatsop.or.us/server/manager

Shortcut to see what services run here: https://delta.co.clatsop.or.us/server/rest/services

If Web Adaptor is not running, you can access it directly via
https://delta.co.clatsop.or.us:6443/arcgis/manager

A data store can be a SQL database or in plain old files (including file geodatabases),
and then the data can be "registered" in ArcGIS Server. Then you can create services
that can be used by Portal to (for example) build web sites and apps.

By default when you install it, it will let you use Database Connections to load and access spatial data.

=== DataStore ===

This component runs on port 2443 is normally hides from you. You access it through server under Sites -> Data Store

Internally it is running couchdb for 3d "scene" storage and postgresql as its relational data store.
It has some arcane hashed system for storing raster tiles in the file system.

See [[ArcGIS DataStore]]

== Admin stuff ==

=== 10.7.1 upgrade notes ===

2019-Oct-24 We did this primarily to get a few feature that will make [[ESRI Collector]] work better. It was time anyway. The features included attachments (photos) and value lists.

Everything went relatively smoothly with 2 exceptions.

Installing Web Adaptor resulting in Error 2878, and I found  
[https://support.esri.com/en/technical-article/000012091 this note]

The installer freaks because we had two bindings for port 443 in IIS. The support note says to remove the second binding for HTTPS (port 443) and then restore it later. In reality a better solution was to delete the FIRST binding and then move on, no need to restore it. This means the server will no longer answer connection requests on http://cc-gis/* but that's okay because there is no certificate for that port anyway.

The other problem came trying to get Portal and Web Adaptor to talk. I tried 50 different combinations of URL and user/pass before finding the one that did, which unfortunately I did not write down. 

One more note, after the upgrade Server you have to install Web Adaptor for Server before you can move ahead in the configuration. That threw me. The Web Adaptor installer will again throw a browser at you with the URL filled in for localhost and the ESRI tech had me change that to the FQDN. I don't know why, but it worked so I don't care. Maybe that was what worked for Portal's Web Adaptor too?

=== Server ===

==== Server not responding 2019-01-09 ====

ESRI case # 02244600 
Travis

ArcGIS Server is supposed to run on ports 6080 and 6443. It's not there. I just checked and "ArcGIS Server" appears to be running in the local services on cc-gis, but when I check to see if the ports are visible, nada. I did this from PowerShell with "netstat -an | findstr 443".

ESRI tech gave me the wordier command '''netstat -aon | findstr "6080 6443 4000 4001 4002 4004 1098 6006 6099"'''

The red "!" next to its name in Portal is a bad sign too. The URL there is https://cc-gis.clatsop.co.clatsop.or.us:6443/arcgis/manager
and none of the variations work either (remove domain, use 6080, etc).

Logging in as CCGISAdmin and as myself did not help either; CCGISAdmin is the "Primary Site Administrator" so it should always work. The login goes through the service. :-)
So if the service is not running... you can't log in. 

We rebooted the machine.

We opened ports on the firewall

<pre>
netsh advfirewall firewall add rule name="Open Server for ArcGIS HTTP" protocol=TCP dir=in localport=6080 action=allow
netsh advfirewall firewall add rule name="Open Server for ArcGIS HTTPS" protocol=TCP dir=in localport=6443 action=allow
netsh advfirewall firewall add rule name="Internally used ports 4000" protocol=TCP dir=in localport=4000 action=allow
netsh advfirewall firewall add rule name="Internally used ports 4001" protocol=TCP dir=in localport=4001 action=allow
netsh advfirewall firewall add rule name="Internally used ports 4002" protocol=TCP dir=in localport=4002 action=allow
netsh advfirewall firewall add rule name="Internally used ports 1098" protocol=TCP dir=in localport=1098 action=allow
netsh advfirewall firewall add rule name="Internally used ports 1099" protocol=TCP dir=in localport=1099 action=allow
netsh advfirewall firewall add rule name="Internally used ports 6006" protocol=TCP dir=in localport=6006 action=allow
netsh advfirewall firewall add rule name="Internally used ports 6099" protocol=TCP dir=in localport=6099 action=allow
</pre>

We gave Full Control to CCGISAdmin on folders in C:\arcgis\arcgisserver

We repaired ArcGIS Server in installed Programs

'''2019-01-14 still broken'''

Switched from user CCGISAdmin to "Local System Account" and restarted the service. No change, still broken. Switched back.

We edited the logging.properties file to enable tomcat log files, and hence discovered the server.xml file was... empty!
Jason is now restoring it.

* C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\conf\ is where the Tomcat config files live.
* C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\log\ is where the Tomcat log files will be once you enable them in log.properties.
* C:/Program Files/ArcGIS/Server/framework/etc/service/logs is where the arcgis server will write its log files, which were not as helpful.

=== Web Adaptor ===

Web Adaptors are ESRI's specialized web proxies. They run on port 80 and redirect traffic to the Portal and Server services.

There is a copy of IIS running on cc-gis, 
and it has 2 copies of Web Adaptor installed in it.

To make changes to the settings in the Web Adaptors, you either need to access via browser from the machine where it's running (cc-gis) or via command line. For us normally this means starting a remote session on cc-gis and then opening one of these URLs:
# http://localhost/arcgis/webadaptor
# http://localhost/portal/webadaptor

''Security note''--
We have it set up so that you access both Portal and ArcGIS Server through Web Adaptor.
Some peeps chose to not allow this for security but we're not planning on exposing our
services directly to the Internet at this time.

=== ArcGIS Data Store ===

This is a separately installed software package, but no one seems to know what it is for. :-)
I do know that it is used as a tile cache so let's go with that. So we will need it.

It runs on port 2443; see https://delta.co.clatsop.or.us:2443/arcgis/datastore/

Getting it set up means connecting it to ArcGIS Server.
That in turn requires connecting Server to Portal, which is called "federating".

In the [https://delta.co.clatsop.or.us:6443/arcgis/manager/site.html ArcGIS Server Manager "Site"] you will be able to see Data Store under Data Stores, click '''VALIDATE ALL'''

[[Image:AGS_manager_Sites.png|500px]]

=== Federation + hosting server ===

'''Federation''' - this means you add the ArcGIS Server to the list of servers available to Portal. In our case, just the one.

# Log into portal, http://delta.co.clatsop.or.us/portal using the primary account (or an admin acct).
# go to Organizations,
# go to EDIT SETTINGS, 
# go to Servers (in left navbar)
# in the Federated Servers section, click Add Server
# Enter services URL https://delta.co.clatsop.or.us/arcgis
# Enter admin URL https://delta.co.clatsop.or.us:6443/arcgis

The process takes about 30 seconds to run and if it works there will be a check mark with 
the admin URL, like this:

[[Image:PortalFederation.png|500px]]

=== Hosting Server ===

# While still on the Servers page, select the newly added server in the drop down list under '''Hosting Servers'''.

There are also options for GeoAnalytics Tools and Raster Analysis.

After the above steps are done, I can see this on the [https://cc-gis:2443/arcgis/datastore/ Data Store URL]:

[[Image:DataStoreOk.png]]

'''UPDATE OCTOBER 2018''' -- I was not able to publish vector tiles from ArcGISPro because the hosting server went away!
I had to call support today, for some reason the setting here went away and Narpat C helped me restore it. It's just a drop down and it had flipped back to "No Hosting Server". '''It has to look like this.'''

'''2019-Oct-23 HOSTING SERVER''' is now https://delta.co.clatsop.or.us/server


[[Image:PortalHostingServer.png]]

<pre>
Hi Brian,

This is Narpat from ESRI Support Services, writing in reference to case #02201305. I have taken ownership of this case.

It was a pleasure talking to you over the phone. To recapitulate our conversation, following are the things we discussed: 
�	Unable to publish Hosted layer's to Portal for ArcGIS.
�	Navigated to Portal for ArcGIS home > Organization > Edit settings > Servers.
�	Set up- the Hosting Server.
I am really glad that we were able to resolve the encountering issue. As discussed, with your permission, I will mark this case as closed for now. However, if you need any assistance in future regarding this case, please contact me back, and I'll be happy to assist you again. 

You can reach me by replying to this email or calling at 888-377-4575 with case #02201305 and your call will be directed to me. 

Have an excellent day ahead!  
</pre>

== Accounts ==

User types described: https://www.esri.com/en-us/arcgis/products/arcgis-online/pricing/arcgis-online-subscriptions#usertypes
In 10.7 supposedly these are available in Portal. All I see are Viewer and Creator which is probably because we need to pay for the others.

Portal access: there is an admin account in the database but all others are from Windows AD.
Normal users get only Viewer access and an account is created the first time they use a service.
We have a license for up to 5 users for Creator access and do more.

Management access to server: use the Primary Site credentials. See the software license page.

To enable Active Directory load this page: https://delta.co.clatsop.or.us/portal/portaladmin/security/config/updateIdentityStore
and put this JSON in.

User store configuration:
 {
  "type": "WINDOWS",
  "properties": {
    "userPassword": "pfkItdaGL2U=",
    "caseSensitive": "false",
    "userEmailAttribute": "mail",
    "user": "clatsop\\winsvc",
    "userFullnameAttribute": "cn",
    "isPasswordEncrypted": "true"
  }
 }

Group store configuration:
 {
  "type": "WINDOWS",
  "properties": {
    "userPassword": "pfkItdaGL2U=",
    "user": "clatsop\\winsvc",
    "isPasswordEncrypted": "true"
  }
 }

Access control to data and services in ArcGIS Server in a federated server (that's what we have here) is also via Portal. 
It is possible to allow all Clatsop County users to have auto-generated accounts to control access.


This is the JSON file that lives here https://delta.co.clatsop.or.us/portal/portaladmin/security/config/update

Try looking here for parameter values: [https://delta.co.clatsop.or.us/portal/portalhelp/apidocs/admin/index.html#/Update_Security_Configuration/032400000019000000/ parameters].
See also the generic imprecise user docs: https://doc.arcgis.com/en/arcgis-online/reference/roles.htm

Originally it was
 {
  "disableServicesDirectory":false,
  "enableAutomaticAccountCreation":false,
  "defaultRoleForUser":"account_user",
  "webgisServerTrustKey":"OarxEyUr53RKGuZiUrn3PiiHtpobAANvv3pYRVf9AhA=",
  "defaultUserTypeIdForUser":""
 }

Now (10/30/2019), (read [https://delta.co.clatsop.or.us/portal/portalhelp/apidocs/admin/index.html#/Security_Configuration/032400000010000000/ API reference pages inside Portal admin])
 {
  "disableServicesDirectory":false,
  "enableAutomaticAccountCreation":'''true''',
  "defaultRoleForUser":"iAAAAAAAAAAAAAAA",
  "webgisServerTrustKey":"OarxEyUr53RKGuZiUrn3PiiHtpobAANvv3pYRVf9AhA=",
  "defaultUserTypeIdForUser":"'''viewerUT'''"
 }

I have no idea why it changed the default role to that. I will update this if I find out!

== Publish print layout templates ==

===Prep===

1. Create a folder on the server [[cc-GIS]] and share it. I used C:/arcgisserver/directories/custom_print_layouts/.<br />
2. Add the folder in ArcGIS Pro Catalog //cc-gis/custom_print_layouts/ so it's easy to find in your project.
[[Image:AGP_catalog.png|500px]]
<br clear=all/>
3. Register the folder in Server as a Data Store. [https://delta.co.clatsop.or.us/server/manager/site.html Server manager -> Sites] -> Data Stores -> Folder
[[Image:RegisterDataStore.png|500px]]
<br clear=all/>

===Publish===

1. Create layout(s) in ArcGIS Pro.<br />
2. Use Share -> Save as Layout file and save them to the custom_print_layouts folder on cc-gis.<br /> 
3. Use the geoprocessing tools to tell server about them.
Now [https://enterprise.arcgis.com/en/server/latest/create-web-apps/windows/tutorial-publish-additional-layouts-for-printing-with-arcgis-pro.htm follow instructions here]

=== Republish ===

After marching through the above steps you can edit and save the page layouts and the changes will be available immediately
via the miracle of the registered data store. If you add or remove layouts you need to go through the publish steps again to update.

In my project I've got a shared folder \\cc-gis\custom_print_layouts that shows up in Catalog under Folders\custom_print_layouts
and I right-click on each layout and use "Save as Layout File".

== Publish a service ==

=== Terminology ===

"Registered data" means the data to be published is in the Enterprise Geodatabase.
"Hosted data" means the data will be copied from its source to the server.

"Registered" is good when the source changes a lot, for example taxlots. Then you don't have to update it.
See [[https://enterprise.arcgis.com/en/server/10.7/publish-services/windows/making-your-data-accessible-to-arcgis-server.htm Make your data accessible] Note your connection file should reference the SDE user and not your Windows user.

"Hosted" is a requirement for some types of publishing, for example cached tiles or vector tiles. It's good
for data that does not change much, for example parks or city limits.

https://enterprise.arcgis.com/en/server/10.6/publish-services/windows/what-is-a-feature-service-.htm

=== Publish from ArcGIS Pro ===

This is just a punch list, details vary widely especially if you are publishing a map versus a feature service.

# Add the feature class to a map tab (in Pro) and set it the way you want (scale ranges, symbology, non-maplex labels)
# Share tab -> Web Layer
# Adjust various and sundry settings
# Click on Analyze
# Fix things flagged in Analyze
# Click on Publish

== Backups ==

''My recollection is that the server things don't install under C:\Program Files but maybe that's because I tested under Linux! :-)''
So I am going with what the docs say.

=== ArcGIS Server ===

[http://enterprise.arcgis.com/en/server/latest/administer/windows/backup-utility.htm ESRI docs] on backup.py

Sample command to backup server 

 python "C:\Program Files\ArcGIS\Server\tools\admin\backup.py -u admin -p mypassword -s http://cc-gis:6080 -f "d:\agsbackups"

Python is probably someplace like C:\Python27\
Please confirm actual location and put it here. ;-)
(and possibly add to PATH?)

Output will be in a file with an .agssite extension in the place you specify with "-f".

==== Restore for Server ====

[http://enterprise.arcgis.com/en/server/latest/administer/windows/back-up-and-restore-your-arcgis-server-site-configuration.htm ESRI Overview] and 
[http://enterprise.arcgis.com/en/server/latest/administer/windows/restore-utility.htm Details on the utility] at ESRI

 <Python installation location>\python.exe "C:\Program Files\ArcGIS\Server\tools\admin\restore.py" -u admin -p mypassword -s http://cc-gis:6080 -f "d:\agsbackups\Jan-24-2013_13-25-23.agssite" -r "d:\agsbackups\log\"

The -r option is not required. Output will go to console either way. You have to create the destination folder.

=== Portal ===

Here is the [http://enterprise.arcgis.com/en/portal/latest/administer/windows/overview-backup-restore-web-gis.htm ESRI overview].

The machine doing the backup has to be able to hit the server at https://cc-gis:6443/arcgis/admin

Copy the properties file
 cd C:\
 mkdir propfiles
 cd propfiles
 cp C:\Program Files\ArcGIS\Portal\tools\webgisdr\webgisdr.properties webgisdr_full.properties

Settings in "full" properties would look like this, more or less.
I wonder if I set SHARED_LOCATION and BACKUP_LOCATION the same what happens?

 BACKUP_STORE_PROVIDER = FileSystem
 SHARED_LOCATION = \\\\cc-gis\\TempPath or some local place for a temporary copy of the backup file.
 BACKUP_LOCATION = \\\\somewhere\\onpremises\\target where the backup will be stored
 PORTAL_ADMIN_URL =
 PORTAL_ADMIN_USERNAME = backups
 PORTAL_ADMIN_PASSWORD = plaintextpassword   (will be encrypted on first run)
 PORTAL_ADMIN_PASSWORD_ENCRYPTED = true
 INCLUDE_SCENE_TILE_CACHES = false
 BACKUP_RESTORE_MODE = full   (or "incremental", make another property file for incremental backups and use one or the other...)

To use webgisdr, we have to enable point-in-time recovery in the data store. 
This procedure does not back up tile caches in Data Store. We should consider the best way to store tiles and back them up.

Run full backups: (maybe add the directory to PATH?)
 cd C:\Program Files\ArcGIS\Portal\tools\webgisdr\
 webgisdr --export --file C:\propfiles\webgisdr_full.properties

If we opted to store anything in files (shapefiles or file geodatabases for example) then those would be backed up and restored
as part of our normal file backups, right?

==== Restore for Portal ====

Recover services etc...

[http://enterprise.arcgis.com/en/portal/latest/administer/windows/restore-web-gis.htm Full ESRI instructions]
 include notes on restoring incrementals, you want to restore the "full" followed by the most recent "incremental".

 cp webgisdr_full.properties recover_full.properties
 edit SHARED_LOCATION to point at the place you have put the backup file
 webgisdr --import --file C:\propfiles\recover_full.properties

Then do the same with the incremental.

Note, we don't plan on using file-based storage at this time, so there won't be files to restore here.

[[Category:ESRI]]
[[Category:GIS]]