Tools that make Windows useful: Difference between revisions

From Wildsong
Jump to navigationJump to search
← Older edit
VisualWikitext
Brian Wilson (talk | contribs)
Bureaucrats, restrict, Administrators, uploaders
11,533 edits
Brian Wilson (talk | contribs)
Bureaucrats, restrict, Administrators, uploaders
11,533 edits
mNo edit summary
 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
I need to be able to work on computers where I have no admin privileges,
Formerly this page was "Portable Windows". Once upon a time I needed to be able to work on computers where I had no admin privileges, couldn't get software installed, and had limited Internet access. Yes! I worked for the federal government!
can't get software installed, and have limited Internet access.


I have developed a kit of tools to help me out. I put them on an external USB drive.
I developed a kit of tools to help me out. When I was a roving code ronin I put them on an external USB drive.
Now that I am a minor bureaucrat I install them on my hard drive. I have risen to the rank of trusted
minor bureaucrat now and have admin rights so over time this page has moved to be just how to install tools.
 
But there is an entire site devoted to the portable concept: https://portableapps.com/
 
I kept everything in one place, a folder called "Portable", the whole folder can be copied a thumb drive.
I have to deal with correctly setting PATH to make it work.
 
== Chocolatey was great ==
 
Then I started using Conda, and using Bash as my shell and using Conda from Bash... and installing bash from "Git For Windows", and all that is fine but then I started using Chocolatey for everything and...
 
But chocolatey requires admin privileges... so portability went out the Windows.
On the other hand, it works on Windows Server, too, so I am happy with it for now.
 
For Conda I decided the miniconda script is better. Go to https://docs.conda.io/en/latest/miniconda.html for the downloader.
 
=== Installing ===
 
Official instructions here https://chocolatey.org/install
 
You copy a string of text and paste it into an elevated PowerShell.
Later on you can upgrade it with itself,
 
choco upgrade chocolatey
 
To run things it installs, add the path C:/ProgramData/chocolatey/bin to the SYSTEM PATH since this will be useable by everyone.
 
You can install whatever you like (in an elevated shell) /SChannel takes care of the SSL certificate error
Here are the things I need. So far it's git, bash, unzip, zip, wget, rsync, conda, emacs, and nodejs.
 
choco install -y git.install --params "/GitAndUnixToolsOnPath /SChannel /WindowsTerminal"
choco install -y unzip zip wget rsync miniconda3
choco install -y emacs
choco install -y nodejs
 
Close your shell window and open a new one in case it changed your PATH settings.
 
For miniconda I had to add C:/tools/miniconda3/Scripts and C:/tools/miniconda3 to my path environment variable.
 
Set CONDA_PATH to C:\tools\miniconda3 and add %CONDA_PATH% and %CONDA_PATH%/Scripts to PATH
 
 
==== Choco has a log! ====
 
You can see what choco commands do by checking its logs in c:/ProgramData/chocolatey/logs
One is verbose and one is a summary, make your choice.
 
== Conda ==
 
Normally I have ArcGIS software installed on Windows and '''I would''' use the Conda environment that comes with it, but at ArcPro 2.8 it stopped working for me in Bash shells. That's why I always install Miniconda3 using Chocolatey. I then make sure it will run the conda of my dreams by changing my bashrc file. In my case I keep the file (called bashrc not .bashrc) in a Github, https://github.com/brian32768/Windows_bin
 
Conda is "Package, dependency and environment management for any language: Python, R, Ruby, Lua, Scala, Java, Javascript, C/ C++, FORTRAN"
 
The full Anaconda installation is huge so I use "mini-conda". "Miniconda is a small �bootstrap� version that includes only conda, Python, and the packages they depend on. "Over 720 scientific packages and their dependencies can be installed individually from the Continuum repository with the �conda install� command." I have not tried using it for anything but Python.
 
I used to use the IDE called Spyder, which is a reasonably complete Python IDE including a debugger. Normally now I use [[Visual Studio Code]].


== GDAL ==
== GDAL ==


Download from gisinternals.com -- this is kept up to date and it has a zip file that you  
I install GDAL as part of Chocolatey but
'''''2020-Mar-09 using GDAL on Windows is madness. Use Docker. Seriously? Windows. Gag me.'''''
 
Download from [http://gisinternals.com gisinternals.com] -- this is kept up to date and it has a zip file that you  
can run binaries from (without admin or installation)
can run binaries from (without admin or installation)


With ArcMap 10.5 I used this:
With ArcGIS Pro 2.5 I used this:
 
DOCKER
 
Back at ArcMap 10.5, this worked:


# Download http://download.gisinternals.com/sdk/downloads/release-1500-x64-gdal-2-1-3-mapserver-7-0-4.zip or the latest zip, I chose the 1500 release because it matches ESRI 10.5 python, not sure if it matters                                                         
# Download http://download.gisinternals.com/sdk/downloads/release-1500-x64-gdal-2-1-3-mapserver-7-0-4.zip or the latest zip, I chose the 1500 release because it matches ESRI 10.5 python, not sure if it matters                                                         
Line 20: Line 83:


There is python stuff in the bin/gdal/python folder which I have not explored yet.
There is python stuff in the bin/gdal/python folder which I have not explored yet.
== GIMP ==
There is a portable version,
https://portableapps.com/apps/graphics_pictures/gimp_portable
I use the standard one now.


== Git and Bash shell ==
== Git and Bash shell ==
Portable Git is awesome.
On Windows 10, for bash and friends there is the [[WSL]] thing but again you have to convince the Man to install it for you.


I rely heavily on GIT to manage Python code, so I have a copy of
I rely heavily on GIT to manage Python code, so I have a copy of
Line 28: Line 102:
You can run the portable version from a thumb drive because does not require admin level installation and it comes with a copy of Bash that also just runs. (Cygwin and GoW for example require admin rights.)
You can run the portable version from a thumb drive because does not require admin level installation and it comes with a copy of Bash that also just runs. (Cygwin and GoW for example require admin rights.)


=== Still missing a little "zip" ===
=== Still missing a little "zip" (and rsync) ===
 
The git environment includes an "unzip" but no "zip" command! To fix this I installed MinGW (in the default C:/MinGW) and then added a .bashrc script in my git bash home. It looks like this:
 
# Additional tools are installed with the MinGW setup program
# and they live here...
PATH=$PATH:/c/MinGW/msys/1.0/bin
export PATH


While I was adding "zip" from MinGW, I also added "rsync" and a few others.
I used to install MinGW but that battled with the Esri version.
This is another place Chocolatey shines; if you can, use it.


Credit: I got the idea for this from https://superuser.com/questions/701141/how-to-add-more-commands-to-git-bash
Credit: I got the idea for this from https://superuser.com/questions/701141/how-to-add-more-commands-to-git-bash


== Conda ==
Short answer, go to http://repo.msys2.org/msys/x86_64/ and download the tools yourself and then install them into /usr/bin/ with "tar".
 
Here are the tar commands. I download from the browser.
Conda is "Package, dependency and environment management for any language: Python, R, Ruby, Lua, Scala, Java, Javascript, C/ C++, FORTRAN"
 
I don't need everything so I use "mini-conda". the packages they depend on. Over 720 scientific packages and their dependencies can be installed individually from the ."


I think I am using the 32-bit Windows version for ESRI compatibility.
Download and install zstd so you can use .zst files.
http://conda.pydata.org/miniconda.html


Using "mini-conda" gets me Spyder, which is a reasonably complete Python IDE
cd /
including a debugger.
tar xJvf /c/Users/bwilson/Downloads/zstd-1.4.7-1-x86_64.pkg.tar.xz


I use a script to set up the environment: conda_setup.bat
Upgrade zstd
and conda_setup.sh


conda_setup.sh
  tar --zstd xvf /c/Users/bwilson/Downloads/zstd-1.4.8-1-x86_64.pkg.tar.zst
emacspath="/e/emacs/bin"
condapath="/e/Miniconda2/Scripts"
PATH="$emacspath:$condapath:$PATH"
export PATH


conda_setup.bat
Download and install rsync and its libraries
set emacspath=E:/emacs/bin
set condapath=E:/Miniconda2/Scripts
setx path=%emacspath%;%condapath%;%path%


=== Spyder ===
tar xv --zstd --file /c/Users/bwilson/Downloads/rsync-3.2.3-1-x86_64.pkg.tar.zst
tar xv --zstd --file /c/Users/bwilson/Downloads/libzstd-1.4.8-1-x86_64.pkg.tar.zst
tar xv --zstd --file /c/Users/bwilson/Downloads/libxxhash-0.8.0-1-x86_64.pkg.tar.zst


See also [[Spyder and ESRI]].
Sometimes I install wget, but then other times I make do with curl.
Download and install zip
tar xvf /c/Users/bwilson/Downloads/zip-3.0-3-x86_64.pkg.tar.xz


After running the Conda script to set the path I can launch spyder.
I open a bash shell and then use


cd /e
== QGIS ==
source conda_setup.sh
spyder


I set spyder to use the ESRI version of Python.
Download the OSGeo4W network installer (64-bit) and run it. When it asks, tell it to install for "Just Me" and not to "Add icon to Start Menu". Optionally, install to your "Portable" folder.


== Emacs ==
[[File:osgeo4wsetup.png]]


Unless you already use it, you probably don't want to learn emacs.
When you get to the "Select Packages" page, select ONLY the version of QGIS.
It's incredibly powerful but obtuse. I have to have it.
For me (today anyway) it's 'qgis-dev', the fragile breaking version because I am in that living-on-the-edge mood.
I download and copy it to my external drive then put set the PATH in
You probably want 'qgis'.
the same scripts that set up Conda.


Download the generic version from https://www.gnu.org/software/emacs/
On the next page, let it install the dependencies.
No need to get a special version, you never install it so you don't need admin rights.


To run it from the Windows GUI, dclick runemacs.exe in the bin/ folder.
Once it's installed it will be found in the Portable folder under OSGEO4W/bin
In a bash shell type "emacs" after setting the PATH.


[[Category: Python]]
[[Category: Python]]
[[Category: GIS]]
[[Category: GIS]]

Latest revision as of 02:08, 17 June 2023

Formerly this page was "Portable Windows". Once upon a time I needed to be able to work on computers where I had no admin privileges, couldn't get software installed, and had limited Internet access. Yes! I worked for the federal government!

I developed a kit of tools to help me out. When I was a roving code ronin I put them on an external USB drive. Now that I am a minor bureaucrat I install them on my hard drive. I have risen to the rank of trusted minor bureaucrat now and have admin rights so over time this page has moved to be just how to install tools.

But there is an entire site devoted to the portable concept: https://portableapps.com/

I kept everything in one place, a folder called "Portable", the whole folder can be copied a thumb drive. I have to deal with correctly setting PATH to make it work.

Chocolatey was great

Then I started using Conda, and using Bash as my shell and using Conda from Bash... and installing bash from "Git For Windows", and all that is fine but then I started using Chocolatey for everything and...

But chocolatey requires admin privileges... so portability went out the Windows. On the other hand, it works on Windows Server, too, so I am happy with it for now.

For Conda I decided the miniconda script is better. Go to https://docs.conda.io/en/latest/miniconda.html for the downloader.

Installing

Official instructions here https://chocolatey.org/install

You copy a string of text and paste it into an elevated PowerShell. Later on you can upgrade it with itself, 

choco upgrade chocolatey

To run things it installs, add the path C:/ProgramData/chocolatey/bin to the SYSTEM PATH since this will be useable by everyone.

You can install whatever you like (in an elevated shell) /SChannel takes care of the SSL certificate error Here are the things I need. So far it's git, bash, unzip, zip, wget, rsync, conda, emacs, and nodejs. 

choco install -y git.install --params "/GitAndUnixToolsOnPath /SChannel /WindowsTerminal"
choco install -y unzip zip wget rsync miniconda3
choco install -y emacs
choco install -y nodejs

Close your shell window and open a new one in case it changed your PATH settings.

For miniconda I had to add C:/tools/miniconda3/Scripts and C:/tools/miniconda3 to my path environment variable.

Set CONDA_PATH to C:\tools\miniconda3 and add %CONDA_PATH% and %CONDA_PATH%/Scripts to PATH


Choco has a log!

You can see what choco commands do by checking its logs in c:/ProgramData/chocolatey/logs One is verbose and one is a summary, make your choice.

Conda

Normally I have ArcGIS software installed on Windows and I would use the Conda environment that comes with it, but at ArcPro 2.8 it stopped working for me in Bash shells. That's why I always install Miniconda3 using Chocolatey. I then make sure it will run the conda of my dreams by changing my bashrc file. In my case I keep the file (called bashrc not .bashrc) in a Github, https://github.com/brian32768/Windows_bin

Conda is "Package, dependency and environment management for any language: Python, R, Ruby, Lua, Scala, Java, Javascript, C/ C++, FORTRAN"

The full Anaconda installation is huge so I use "mini-conda". "Miniconda is a small �bootstrap� version that includes only conda, Python, and the packages they depend on. "Over 720 scientific packages and their dependencies can be installed individually from the Continuum repository with the �conda install� command." I have not tried using it for anything but Python.

I used to use the IDE called Spyder, which is a reasonably complete Python IDE including a debugger. Normally now I use Visual Studio Code.

GDAL

I install GDAL as part of Chocolatey but 2020-Mar-09 using GDAL on Windows is madness. Use Docker. Seriously? Windows. Gag me.

Download from gisinternals.com -- this is kept up to date and it has a zip file that you can run binaries from (without admin or installation)

With ArcGIS Pro 2.5 I used this:

DOCKER

Back at ArcMap 10.5, this worked:

  1. Download http://download.gisinternals.com/sdk/downloads/release-1500-x64-gdal-2-1-3-mapserver-7-0-4.zip or the latest zip, I chose the 1500 release because it matches ESRI 10.5 python, not sure if it matters
  2. Unpack it someplace convenient, I put it in my home directory under GDAL, so mkdir C:/Users/bwilson/GDAL
  3. Add it to your PATH, I added this to the end C:\Users\bwilson\GDAL\bin;C:\Users\bwilson\GDAL\bin\gdal\apps
  4. If it worked then you should be able to open a cmd window and type 'ogrinfo', and get usage info not an error.

This does NOT install any Python packages, it just gets me the GDAL command line programs, which I can call using Python "subprocess" calls. It also gets me a copy of mapserver, for which I have no use at this time.

There is python stuff in the bin/gdal/python folder which I have not explored yet.

GIMP

There is a portable version, https://portableapps.com/apps/graphics_pictures/gimp_portable I use the standard one now.

Git and Bash shell

Portable Git is awesome.

On Windows 10, for bash and friends there is the WSL thing but again you have to convince the Man to install it for you.

I rely heavily on GIT to manage Python code, so I have a copy of "PortableGit". https://www.nuget.org/packages/PortableGit/

You can run the portable version from a thumb drive because does not require admin level installation and it comes with a copy of Bash that also just runs. (Cygwin and GoW for example require admin rights.)

Still missing a little "zip" (and rsync)

I used to install MinGW but that battled with the Esri version. This is another place Chocolatey shines; if you can, use it.

Credit: I got the idea for this from https://superuser.com/questions/701141/how-to-add-more-commands-to-git-bash

Short answer, go to http://repo.msys2.org/msys/x86_64/ and download the tools yourself and then install them into /usr/bin/ with "tar". Here are the tar commands. I download from the browser.

Download and install zstd so you can use .zst files. 

cd /
tar xJvf /c/Users/bwilson/Downloads/zstd-1.4.7-1-x86_64.pkg.tar.xz

Upgrade zstd 

 tar --zstd xvf /c/Users/bwilson/Downloads/zstd-1.4.8-1-x86_64.pkg.tar.zst

Download and install rsync and its libraries 

tar xv --zstd --file /c/Users/bwilson/Downloads/rsync-3.2.3-1-x86_64.pkg.tar.zst
tar xv --zstd --file /c/Users/bwilson/Downloads/libzstd-1.4.8-1-x86_64.pkg.tar.zst
tar xv --zstd --file /c/Users/bwilson/Downloads/libxxhash-0.8.0-1-x86_64.pkg.tar.zst

Sometimes I install wget, but then other times I make do with curl.

Download and install zip 

tar xvf /c/Users/bwilson/Downloads/zip-3.0-3-x86_64.pkg.tar.xz


QGIS

Download the OSGeo4W network installer (64-bit) and run it. When it asks, tell it to install for "Just Me" and not to "Add icon to Start Menu". Optionally, install to your "Portable" folder.

When you get to the "Select Packages" page, select ONLY the version of QGIS. For me (today anyway) it's 'qgis-dev', the fragile breaking version because I am in that living-on-the-edge mood. You probably want 'qgis'.

On the next page, let it install the dependencies.

Once it's installed it will be found in the Portable folder under OSGEO4W/bin

Retrieved from "https://wiki.wildsong.biz//index.php?title=Tools_that_make_Windows_useful&oldid=13825"