Tools that make Windows useful: Difference between revisions

From Wildsong
Jump to navigationJump to search
← Older edit
VisualWikitext
Brian Wilson (talk | contribs)
Bureaucrats, restrict, Administrators, uploaders
11,540 edits
Brian Wilson (talk | contribs)
Bureaucrats, restrict, Administrators, uploaders
11,540 edits
mNo edit summary
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
I need to be able to work on computers where I have no admin privileges,
Formerly this page was "Portable Windows". Once upon a time I needed to be able to work on computers where I had no admin privileges, couldn't get software installed, and had limited Internet access. Yes! I worked for the federal government!
can't get software installed, and have limited Internet access.


I have developed a kit of tools to help me out. When I was a roving code ronin I put them on an external USB drive.
I developed a kit of tools to help me out. When I was a roving code ronin I put them on an external USB drive.
Now that I am a minor bureaucrat I install them on my hard drive (still all done without admin privileges.)
Now that I am a minor bureaucrat I install them on my hard drive. I have risen to the rank of trusted
minor bureaucrat now and have admin rights so over time this page has moved to be just how to install tools.


There is an entire site devoted to this concept, it turns out: https://portableapps.com/
But there is an entire site devoted to the portable concept: https://portableapps.com/


I've started keeping everything in one place, a folder called "Portable", I can copy the whole folder to a thumb drive.
I kept everything in one place, a folder called "Portable", the whole folder can be copied a thumb drive.
I have to deal with correctly setting PATH to make it work.
I have to deal with correctly setting PATH to make it work.


Probably following the PortableApps.com page would be the way to go, I have to look at it more.
== Chocolatey was great ==


== Conda ==
Then I started using Conda, and using Bash as my shell and using Conda from Bash... and installing bash from "Git For Windows", and all that is fine but then I started using Chocolatey for everything and...


'''update 2/2/2021''' -- normally I have ArcGIS software installed on Windows and use the Conda environment that comes with it.
But chocolatey requires admin privileges... so portability went out the Windows.
When working on an ArcGIS machine I do NOT install Miniconda3, it really gets confusing if I do. Instead I update the outdated version of Conda
On the other hand, it works on Windows Server, too, so I am happy with it for now.
that Esri provides. If you try to update conda it will tell you what to do, next time I go through that I need to write it down.
The big reason for updating is that I use Visual Studio Code and it expects something more recent.


Conda is "Package, dependency and environment management for any language: Python, R, Ruby, Lua, Scala, Java, Javascript, C/ C++, FORTRAN"
For Conda I decided the miniconda script is better. Go to https://docs.conda.io/en/latest/miniconda.html for the downloader.


=== Installing ===


I use the 64-bit version most of the time and I use the 32-bit Windows version for ESRI compatibility when I need to access "Personal Geodatabases" (mdb files).
Official instructions here https://chocolatey.org/install
http://conda.pydata.org/miniconda.html


I used to use the IDE called Spyder, which is a reasonably complete Python IDE including a debugger. Normally now I use Visual Studio Code.
You copy a string of text and paste it into an elevated PowerShell.  
Later on you can upgrade it with itself,


From an external drive E: I run these scripts to set up the environment when I arrive on site: conda_setup.bat
choco upgrade chocolatey
and conda_setup.sh


conda_setup.sh
To run things it installs, add the path C:/ProgramData/chocolatey/bin to the SYSTEM PATH since this will be useable by everyone.
emacspath="/e/emacs/bin"
condapath="/e/Miniconda2/Scripts"
PATH="$emacspath:$condapath:$PATH"
export PATH


conda_setup.bat
You can install whatever you like (in an elevated shell) /SChannel takes care of the SSL certificate error
set emacspath=E:/emacs/bin
Here are the things I need. So far it's git, bash, unzip, zip, wget, rsync, conda, emacs, and nodejs.
set condapath=E:/Miniconda2/Scripts
setx path=%emacspath%;%condapath%;%path%


=== Spyder ===
choco install -y git.install --params "/GitAndUnixToolsOnPath /SChannel /WindowsTerminal"
choco install -y unzip zip wget rsync miniconda3
choco install -y emacs
choco install -y nodejs


See also [[Spyder and ESRI]].
Close your shell window and open a new one in case it changed your PATH settings.


After running the Conda script to set the path I can launch spyder.
For miniconda I had to add C:/tools/miniconda3/Scripts and C:/tools/miniconda3 to my path environment variable.
I open a bash shell and then use


cd /e
Set CONDA_PATH to C:\tools\miniconda3 and add %CONDA_PATH% and %CONDA_PATH%/Scripts to PATH
source conda_setup.sh
spyder


I normally set spyder to use the ESRI installed version of Python.


== Emacs ==
==== Choco has a log! ====


Unless you already use it, you probably don't want to learn emacs.
You can see what choco commands do by checking its logs in c:/ProgramData/chocolatey/logs
It's incredibly powerful but obtuse. I have to have it.
One is verbose and one is a summary, make your choice.
I download and copy it to my external drive then set the PATH in
the same scripts that set up Conda.


Download the generic version from https://www.gnu.org/software/emacs/
== Conda ==
No need to get a special version, you never install it so you don't need admin rights.


To run it from the Windows GUI, dclick runemacs.exe in the bin/ folder.
Normally I have ArcGIS software installed on Windows and '''I would''' use the Conda environment that comes with it, but at ArcPro 2.8 it stopped working for me in Bash shells. That's why I always install Miniconda3 using Chocolatey. I then make sure it will run the conda of my dreams by changing my bashrc file. In my case I keep the file (called bashrc not .bashrc) in a Github, https://github.com/brian32768/Windows_bin
In a bash shell type "emacs" after setting the PATH.
 
Conda is "Package, dependency and environment management for any language: Python, R, Ruby, Lua, Scala, Java, Javascript, C/ C++, FORTRAN"
 
The full Anaconda installation is huge so I use "mini-conda". "Miniconda is a small �bootstrap� version that includes only conda, Python, and the packages they depend on. "Over 720 scientific packages and their dependencies can be installed individually from the Continuum repository with the �conda install� command." I have not tried using it for anything but Python.
 
I used to use the IDE called Spyder, which is a reasonably complete Python IDE including a debugger. Normally now I use [[Visual Studio Code]].


== GDAL ==
== GDAL ==


'''''2020-Mar-09 Using GDAL on Windows is madness. Use Docker. Seriously? Windows. Gag me.'''''
I install GDAL as part of Chocolatey but
'''''2020-Mar-09 using GDAL on Windows is madness. Use Docker. Seriously? Windows. Gag me.'''''


Download from [http://gisinternals.com gisinternals.com] -- this is kept up to date and it has a zip file that you  
Download from [http://gisinternals.com gisinternals.com] -- this is kept up to date and it has a zip file that you  
Line 93: Line 86:
== GIMP ==
== GIMP ==


There is a portable version,
https://portableapps.com/apps/graphics_pictures/gimp_portable
https://portableapps.com/apps/graphics_pictures/gimp_portable
I use the standard one now.


== Git and Bash shell ==
== Git and Bash shell ==
Portable Git is awesome.


On Windows 10, for bash and friends there is the [[WSL]] thing but again you have to convince the Man to install it for you.
On Windows 10, for bash and friends there is the [[WSL]] thing but again you have to convince the Man to install it for you.
Line 104: Line 102:
You can run the portable version from a thumb drive because does not require admin level installation and it comes with a copy of Bash that also just runs. (Cygwin and GoW for example require admin rights.)
You can run the portable version from a thumb drive because does not require admin level installation and it comes with a copy of Bash that also just runs. (Cygwin and GoW for example require admin rights.)


=== Still missing a little "zip" ===
=== Still missing a little "zip" (and rsync) ===


The git environment includes an "unzip" but no "zip" command! To fix this I installed [http://www.mingw.org/wiki/Getting_Started MinGW] (in the default C:/MinGW) and then added a .bashrc script in my git bash home. It looks like this:
I used to install MinGW but that battled with the Esri version.  
This is another place Chocolatey shines; if you can, use it.


# Additional tools are installed with the MinGW setup program
Credit: I got the idea for this from https://superuser.com/questions/701141/how-to-add-more-commands-to-git-bash
# and they live here...
 
  PATH=$PATH:/c/MinGW/msys/1.0/bin
Short answer, go to http://repo.msys2.org/msys/x86_64/ and download the tools yourself and then install them into /usr/bin/ with "tar".
export PATH
Here are the tar commands. I download from the browser.
 
Download and install zstd so you can use .zst files.
 
cd /
  tar xJvf /c/Users/bwilson/Downloads/zstd-1.4.7-1-x86_64.pkg.tar.xz
 
Upgrade zstd
 
  tar --zstd xvf /c/Users/bwilson/Downloads/zstd-1.4.8-1-x86_64.pkg.tar.zst
 
Download and install rsync and its libraries


While I was adding "zip" from MinGW, I also added "rsync" and a few others.
tar xv --zstd --file /c/Users/bwilson/Downloads/rsync-3.2.3-1-x86_64.pkg.tar.zst
tar xv --zstd --file /c/Users/bwilson/Downloads/libzstd-1.4.8-1-x86_64.pkg.tar.zst
tar xv --zstd --file /c/Users/bwilson/Downloads/libxxhash-0.8.0-1-x86_64.pkg.tar.zst


Credit: I got the idea for this from https://superuser.com/questions/701141/how-to-add-more-commands-to-git-bash
Sometimes I install wget, but then other times I make do with curl.
Download and install zip
tar xvf /c/Users/bwilson/Downloads/zip-3.0-3-x86_64.pkg.tar.xz


== Node.js ==


== QGIS ==
== QGIS ==


Download the OSGeo4W network installer (64-bit) and run it. When it asks, tell it to install for "Just Me" and not to "Add icon to Start Menu". Install to your "Portable" folder.
Download the OSGeo4W network installer (64-bit) and run it. When it asks, tell it to install for "Just Me" and not to "Add icon to Start Menu". Optionally, install to your "Portable" folder.


[[File:osgeo4wsetup.png]]
[[File:osgeo4wsetup.png]]

Latest revision as of 02:08, 17 June 2023

Formerly this page was "Portable Windows". Once upon a time I needed to be able to work on computers where I had no admin privileges, couldn't get software installed, and had limited Internet access. Yes! I worked for the federal government!

I developed a kit of tools to help me out. When I was a roving code ronin I put them on an external USB drive. Now that I am a minor bureaucrat I install them on my hard drive. I have risen to the rank of trusted minor bureaucrat now and have admin rights so over time this page has moved to be just how to install tools.

But there is an entire site devoted to the portable concept: https://portableapps.com/

I kept everything in one place, a folder called "Portable", the whole folder can be copied a thumb drive. I have to deal with correctly setting PATH to make it work.

Chocolatey was great

Then I started using Conda, and using Bash as my shell and using Conda from Bash... and installing bash from "Git For Windows", and all that is fine but then I started using Chocolatey for everything and...

But chocolatey requires admin privileges... so portability went out the Windows. On the other hand, it works on Windows Server, too, so I am happy with it for now.

For Conda I decided the miniconda script is better. Go to https://docs.conda.io/en/latest/miniconda.html for the downloader.

Installing

Official instructions here https://chocolatey.org/install

You copy a string of text and paste it into an elevated PowerShell. Later on you can upgrade it with itself, 

choco upgrade chocolatey

To run things it installs, add the path C:/ProgramData/chocolatey/bin to the SYSTEM PATH since this will be useable by everyone.

You can install whatever you like (in an elevated shell) /SChannel takes care of the SSL certificate error Here are the things I need. So far it's git, bash, unzip, zip, wget, rsync, conda, emacs, and nodejs. 

choco install -y git.install --params "/GitAndUnixToolsOnPath /SChannel /WindowsTerminal"
choco install -y unzip zip wget rsync miniconda3
choco install -y emacs
choco install -y nodejs

Close your shell window and open a new one in case it changed your PATH settings.

For miniconda I had to add C:/tools/miniconda3/Scripts and C:/tools/miniconda3 to my path environment variable.

Set CONDA_PATH to C:\tools\miniconda3 and add %CONDA_PATH% and %CONDA_PATH%/Scripts to PATH


Choco has a log!

You can see what choco commands do by checking its logs in c:/ProgramData/chocolatey/logs One is verbose and one is a summary, make your choice.

Conda

Normally I have ArcGIS software installed on Windows and I would use the Conda environment that comes with it, but at ArcPro 2.8 it stopped working for me in Bash shells. That's why I always install Miniconda3 using Chocolatey. I then make sure it will run the conda of my dreams by changing my bashrc file. In my case I keep the file (called bashrc not .bashrc) in a Github, https://github.com/brian32768/Windows_bin

Conda is "Package, dependency and environment management for any language: Python, R, Ruby, Lua, Scala, Java, Javascript, C/ C++, FORTRAN"

The full Anaconda installation is huge so I use "mini-conda". "Miniconda is a small �bootstrap� version that includes only conda, Python, and the packages they depend on. "Over 720 scientific packages and their dependencies can be installed individually from the Continuum repository with the �conda install� command." I have not tried using it for anything but Python.

I used to use the IDE called Spyder, which is a reasonably complete Python IDE including a debugger. Normally now I use Visual Studio Code.

GDAL

I install GDAL as part of Chocolatey but 2020-Mar-09 using GDAL on Windows is madness. Use Docker. Seriously? Windows. Gag me.

Download from gisinternals.com -- this is kept up to date and it has a zip file that you can run binaries from (without admin or installation)

With ArcGIS Pro 2.5 I used this:

DOCKER

Back at ArcMap 10.5, this worked:

  1. Download http://download.gisinternals.com/sdk/downloads/release-1500-x64-gdal-2-1-3-mapserver-7-0-4.zip or the latest zip, I chose the 1500 release because it matches ESRI 10.5 python, not sure if it matters
  2. Unpack it someplace convenient, I put it in my home directory under GDAL, so mkdir C:/Users/bwilson/GDAL
  3. Add it to your PATH, I added this to the end C:\Users\bwilson\GDAL\bin;C:\Users\bwilson\GDAL\bin\gdal\apps
  4. If it worked then you should be able to open a cmd window and type 'ogrinfo', and get usage info not an error.

This does NOT install any Python packages, it just gets me the GDAL command line programs, which I can call using Python "subprocess" calls. It also gets me a copy of mapserver, for which I have no use at this time.

There is python stuff in the bin/gdal/python folder which I have not explored yet.

GIMP

There is a portable version, https://portableapps.com/apps/graphics_pictures/gimp_portable I use the standard one now.

Git and Bash shell

Portable Git is awesome.

On Windows 10, for bash and friends there is the WSL thing but again you have to convince the Man to install it for you.

I rely heavily on GIT to manage Python code, so I have a copy of "PortableGit". https://www.nuget.org/packages/PortableGit/

You can run the portable version from a thumb drive because does not require admin level installation and it comes with a copy of Bash that also just runs. (Cygwin and GoW for example require admin rights.)

Still missing a little "zip" (and rsync)

I used to install MinGW but that battled with the Esri version. This is another place Chocolatey shines; if you can, use it.

Credit: I got the idea for this from https://superuser.com/questions/701141/how-to-add-more-commands-to-git-bash

Short answer, go to http://repo.msys2.org/msys/x86_64/ and download the tools yourself and then install them into /usr/bin/ with "tar". Here are the tar commands. I download from the browser.

Download and install zstd so you can use .zst files. 

cd /
tar xJvf /c/Users/bwilson/Downloads/zstd-1.4.7-1-x86_64.pkg.tar.xz

Upgrade zstd 

 tar --zstd xvf /c/Users/bwilson/Downloads/zstd-1.4.8-1-x86_64.pkg.tar.zst

Download and install rsync and its libraries 

tar xv --zstd --file /c/Users/bwilson/Downloads/rsync-3.2.3-1-x86_64.pkg.tar.zst
tar xv --zstd --file /c/Users/bwilson/Downloads/libzstd-1.4.8-1-x86_64.pkg.tar.zst
tar xv --zstd --file /c/Users/bwilson/Downloads/libxxhash-0.8.0-1-x86_64.pkg.tar.zst

Sometimes I install wget, but then other times I make do with curl.

Download and install zip 

tar xvf /c/Users/bwilson/Downloads/zip-3.0-3-x86_64.pkg.tar.xz


QGIS

Download the OSGeo4W network installer (64-bit) and run it. When it asks, tell it to install for "Just Me" and not to "Add icon to Start Menu". Optionally, install to your "Portable" folder.

When you get to the "Select Packages" page, select ONLY the version of QGIS. For me (today anyway) it's 'qgis-dev', the fragile breaking version because I am in that living-on-the-edge mood. You probably want 'qgis'.

On the next page, let it install the dependencies.

Once it's installed it will be found in the Portable folder under OSGEO4W/bin

Retrieved from "https://wiki.wildsong.biz//index.php?title=Tools_that_make_Windows_useful&oldid=13825"