Working with git

Overview

Most work I do is out on github in public repos that anyone can access, search for geo-ceg and brian32768. I can't put every bit of my life out on the Internet, so I still operate my own Git server for proprietary projects. I also work in other peoples' servers for projects.

This page contains notes both on Github and on using private Git servers.

I found that chapter 3 of the book "Building Enterprise JavaScript Applications" contains a pretty succinct introduction to git.

Git commands that I use most often

The main ones, obviously: init, clone, add, commit, status, push, pull (instead of pull everyone SHOULD use fetch and merge)

The ones I forget about: log

Until today I never bothered with branches (missing half the fun!) because 99.9% of the time right now I work on code by myself. This is a stupid excuse since it means having a broken master branch on Github most of the time as I do nightly checkpoints to switch between home and work. So, read the section below on branches.

I should do more tagging.

Recovering from staging a file by accident:

git add oops.cs
git reset HEAD oops.cs

"Master" -> "Main"

At this point I have some repos with "master" as the default branch, some with "main", and some with "release". Over time they will migrate to "main". Randomly. Yeesh.

2021-02 - It's possible to simply rename a master branch in github now. They opted for "main".

Changing "master" to "main"

Recently github added rename ability to make moving to "main" easier. You can still do it this way though. These instructions I found on Stack Overflow and modified to suit my own use.

git commit -m "Commit any pending changes." -a
git push
git branch main           # create branch
git checkout main         # switch to the new branch
git push -u origin main   # push the branch to the remote and track it
git branch -d master      # delete local master

Go to the repository at https://github.com/ and bring up the Settings page. Go to Branches and make "main" the main branch.

git push --delete origin master   # delete remote master
git remote prune origin           # delete the remote tracking branch

git checkout -b dev

Now get on with life, working and testing in "dev" and merging into "main".

Setting execute bit

After the file is already in the repo,

git update-index --chmod=+x path/to/file

On add (a better way IMO),

git add --chmod=+x path/to/file

Using update-index is awkward because you then have to change the file before you can commit the +x.

Github workflow with "main" and "dev" branches

I am (from now on) following "the Driessen model". Read this carefully, especially regarding feature branches, which I started using today. :-) Or maybe soon. It's well written and there is no point in reinventing it here. HOWEVER there is value for me in running through my own tests here so I can remember what I did!

All day-to-day work is done in the "dev" branch. After confirming everything works in "dev", changes are merged into "main" branch and I tag it as a release.

Keep in mind that Git is distributed so a branch can exist on your local machine and never touch the repository. The entire branch lifecycle can stay on your local hard drive if you don't need to share (or move between two computers the way I do every day). Alternatively, you can have a branch track a remote repository.

The excellent free Pro Git book on the Git site explains branching in more detail including getting rid of a branch and dealing with merge conflicts.

Here is another explanation about the nuts and bolts of merging. http://longair.net/blog/2009/04/16/git-fetch-and-merge/

Normal project workflow

If "dev" code not working, don't do merges to main branch, duh, it's for releases!

Let's try all this out.

Create repo

I am using my personal github account "brian32768", you use your own. :-)

Create a new empty github project called "brian32768/gt". (I let it add README.md and LICENSE.txt) Before going further, make sure it's "main" and not "master", and create a "dev" branch, and if you need to, make "main" the default, and delete "master".

Create a local, working copy.

cd ~/source/repos
git clone [email protected]:brian32768/gt.git
cd gt

Create a development branch "dev".

git checkout -b dev

Show the branch we're on; there will be two now "dev" and "main". "git status" shows the branch, too.

git branch

Henceforth do all development and testing on "dev"

Edit the README.md

vi README.md

Add a new file.

vi index.html
git add index.html

Stage the changes and push them up to github

git commit -m 'here we go a coding, ra ra ra' -a
git push --set-upstream origin dev

(Just the usual "git push" is not good enough the first push, you have to use set-upstream to create the branch in the repo.)

If you look in github now you should see both "main" and "dev" branches. The "main" branch should be untouched at this time.

Work in another place

I should be able to go home, clone on Murre, move to the "dev" branch, and make changes there. (Or I could just chdir to a separate place on my work computer, to mimic this workflow. :-)

Assuming that I've created a dev branch as in the previous section,

git clone [email protected]:brian32768/gt.git gt_dev2
cd gt_dev2
git checkout -b dev origin/dev

Leave off "origin/dev" and it will create a new local branch called "dev", and this can be confusing! You can do this when you are on the local dev branch, and this will push the code to github and create a new branch there at the same time.

 git push --set-upstream origin dev

Now I can see my typo when I look at README.md so I know I am in the right place.

vi README.md
git commit -m "made a change at home"
git push

Now when I go back to computer #1 I need to get any changes I made at home, since both working directories (at work and at home) are on the "dev" branch, this should work.

cd source/repos/gt
git fetch

There should not be any changes on the branch to merge so no 'git merge' needed here.

Now check contents of README.md to make sure the change came in correctly.

Merge dev changes into main

When you think you have working code, you must merge dev changes back into main.

Assuming you are currently on the dev branch, stage any changes and push them.

git commit -a -m 'good to go'
git push

Switch to the main branch

git checkout main

Pull changes from dev branch into main

git merge dev

and push

git push

Tag

git tag -a "namespaces" -m "merged new javascript_patterns namespaces code from dev"

Jump back to the dev branch and continue working.

git checkout dev

Revert a change

For the purposes of my test, let's say that the recent change was wrong and I want to revert main to the previous version. Find the commit; if you tagged this is the same as using the tag.

git log
commit a5b7b10caac48a5514ce781c28b71846ff8d1e5b (HEAD -> dev, tag: 1.0, origin/dev)
Author: Brian Wilson <[email protected]>
Date:   Wed Nov 7 16:24:10 2018 -0800

   added logo

commit d7a26f4c868bf29c749600f62386521d35a3044b
Author: Brian Wilson <[email protected]>
Date:   Wed Nov 7 16:04:12 2018 -0800

   typo fixed

(Ugh, did not mean to add the logo. Previous commit was "d7a26f4c868bf29c749600f62386521d35a3044b")

BRIAN! CONTINUE WRITING HERE Brian Wilson (talk) 01:00, 8 November 2018 (UTC)

Now I am back at the previous version in main. I can leave things this way indefinitely while I deal with the issue in "dev", or I can do the same thing in "dev" - revert to the previous version to get rid of the broken changes. In this demo I just push ahead.

So I fix "dev". Then, I check in the changes on "dev" and merge to "main" again. This time the change is fine.

Feature branching

"Dev" is just the normal day to day stuff, and when I create a feature branch it's off of "dev". When I create a branch to support a specific production version that uses different assets (for example special branding), that's branched directly off "main".

Working with variants

The other use case I want to test is how to work with a variant branch; in this case, I want to create two different web sites with different branding.

To test this, I add the generic "map46.svg" and accompanying CSS style sheet to "main".

I create the variant branch and add a Clatsop County ""cclogo.gif" logo there with a different style sheet that will insert this graphic.

Now the question is, how do I support both? I make a change to "dev", merge it back to "main", and then I have to merge changes from "main" to variant without stepping on the assets. Since the assets did not change on "main", this should be no problem.

If I do change the (shared) style sheet for example, then I will have to do a more careful merge.

Pull requests

Rebase

I have an outdated fork and need to pull in changes from the original repo before I can send a new pull request.

git remote add upstream original repository
git fetch upstream
git checkout main
git rebase upstream/main

Making some changes

git checkout -b env_doc_edit
edit the file(s)
commit changes

Migrating git projects to Github.com

I have migrated most of my shareable projects to github.com to get them off my own server, making them more accessible and creating an off-site backup at no cost to me. Pretty much all the same git commands below work.

Migration: Use the Github Importer.

I used the command line import method. Basically you pull the repo with all its history onto your local machine then push it up to github. In brief

Create the repository first at github, "ssurgo" in this case.
Make a clone of your existing repo: git clone --bare [email protected]:ssurgo.git
cd ssurgo
Push everything up to github: git push --mirror https://github.com/geo-ceg/ssurgo.git
cd ..
Make sure it worked by cloning it! git clone [email protected]:Geo-CEG/ssurgo.git
If it worked you have a directory "ssurgo" now. Remove the bare repo: rm -rf ssurgo.git
Try commiting by making a README. emacs README.me
git add README.md
git commit -a -m "added README!"
git push

Add Github repositories from the command line

conda create hub
conda activate hub
conda install hub

for i in `cat folders`; do
 cd $i
 git init -b main
 git add .
 git commit -m initial\ commit -a
 hub create -d "A widget from Esri \"Web AppBuilder, Developer Edition\"." Wildsong/wabde-widget-$i.git
 git remote -v
 git push origin HEAD

for i in `cat folders`; do
 git tag
 touch README.md
 echo "= $i" >> README.md
 cat ../README.md >> README.md
 cp ../LICENSE .
 git add README.md LICENSE
 git commit -m 'added generic README and LICENSE files.'
 git push

Use case: you moved a repo from a personal account to an organization account

You can do that on github.com but now you need to change the repo, actually you don't have to but it's tidier.*

git remote set-url origin newurl

If you don't change it, github leaves a link behind to redirect things and it will all be fine.

Use case: git for web app development

One of the main reasons I want my own git server is so that I can sync a web app between my laptop and a server.

Work flow:

Develop and test test test on the laptop, in the "dev" branch (or in a "feature" branch").
When you think it's flawless, push changes to the git server, still on the "dev" branch.
On the web server, move onto the dev branch and pull the changes down.
Test live server. If the live server breaks after an update, switch back to "main" and pull to revert.
After confirming web site is working, merge changes into "main", tag, and push.
On web server, pull "main" down to update it.

Authentication

At work I have to use a standalone git server that uses Active Directory and I also use github.

On Windows, I install the git-scm ("Git for Windows") package and use its default git-credential authentication code. https://git-scm.com/docs/git-credential

I set up a file called "cred" and fed it to git, the file contains

protocol=https
host=MYGITHOSTNAME like codebase.domain.com
username=MYNAME
password=MYPASSWORD

then I did

git credential approve < cred

I then tested a clone operation and it did not prompt for credentials so I am happy. Well, except I don't know where it keeps the creds on Windows. I moved "cred" and tried again and the command still works.

Debian

Refer to https://www.shellhacks.com/git-config-username-password-store-credentials/

On Debian, first I told it to use a global settings file (~/.git-credentials) by typing

git config --global credential.helper store

I used a git clone command and it stored the credentials after I successfully connected one time.

If it's not working you can reconnect with (or connect the first time)

gh auth login

and select the web browser option even if you are in an ssh shell -- it will fail and then reveal the URL so you can paste it into a browser.

This will give you a code and then send you to https://github.com/login/device. You enter that code into your browser. I guess it saves the link between the device and you in Github somewhere.

Well, this confirms you can connect from your desktop not that shell remote server. Maybe it does not matter since we have just one IP address at work.

Maybe you need a new personal access token, they expire. In Github look in Settings->Developer Settings (way down at the bottom) -> Personal Access Tokens -> Classic

Running a git server

File ownerships and ssh access control (which I no longer use.)

Access to the server is through ssh. The repositories will be owned by the user 'git'. So I put the home for user 'git' where I want the repositories to be; on Bellman I use /green/repositories. Then I set up ssh with keys so that I don't need to share passwords.

I put create a separate 'git' key pair and put the public key into ~git/.ssh/authorized_keys I can test it with "ssh [email protected] list", this should list the available repos and exit.

Once that's all in place I don't need a complete path for the "remote" command, just one relative to git's home.

Here is how I generate a key for github

ssh-keygen -t rsa -b 4096 -C my_email_address

eval $(ssh-agent -s) 
ssh-add ~/.ssh/id_rsa_github

Here is a quick way to test it

ssh [email protected]
.
.
.
Hi brian32768! You've successfully authenticated, but GitHub does not provide shell access.

Creating a new private repo

I am using ssh for file transfer.

I find it easiest to create a new empty repo directly on the server and then push existing content into it.

ssh bellman
cd ~git
sudo git init --bare myproject.git create a bare repository
sudo chown -R git.wildsong myproject.git
exit

cd source/repos
assuming myproject is a folder containing the files you want in a repo
cd myproject
git init
make a gitignore file in myproject
git add some files
git commit -m 'initial commit'
git remote add origin [email protected]:myproject.git
git push origin main this pushes committed files from the local repository to the "main" branch on the remote machine

Now I should be able to clone my project onto the web server and start using git to keep it updated.

ssh webserver
cd /var/www/appserver
git clone git.wildsong.biz:myproject.git

On my copy I need to define the remote repo,

git remote add origin [email protected]:myproject.git

When I make changes on the local laptop, first I commit them locally.

git commit

..then I push them up to the git server

git push origin main

..then I pull them down onto the public web server

git fetch

Then I create a virtualenv and load the requirements.

cd myproject
virtualenv env
source env/bin/activate
pip install -r requirements.txt
python run.py

Github collaboration

Now I want to contribute to someone's project. This description is good especially after reading the comments section too.

http://blog.davidecoppola.com/2016/11/howto-contribute-to-open-source-project-on-github/

Briefly, leaving out some critical steps...

Fork -- Makes a copy of the project in Github that I can change
Clone -- Pull the code down from Github to my computer
Branch -- I will work in a branch and then remove it when done
Edit -- Make proposed changes
Issue pull request -- Notifies project owner; hopefully it will be accepted

Git submodules

I keep some code in github, and I want to incorporate it into a project. I can use the submodule feature.

cd myproject
git submodule add [email protected]:brian32768/pyst2.git

This clones pyst2 into a subdirectory (called pyst2 by default) Now I can work on pyst2 and do commits and pushes but it remains separate from 'myproject'. If I ask for status...

git status
On branch "main"
Your branch is up-to-date with 'origin/main'.
Changes to be committed:
 (use "git reset HEAD <file>..." to unstage)

modified: .gitmodules new file: pyst2

When you clone a new copy or update an existing clone (git pull) then the submodules will be there but empty, you have do do this

git submodule update --init

That will tell git to populate each submodule folder with code from the other repository.

I am not sure why but it left the HEAD detached so I had to do

git checkout main

Git Server in Docker

This would be to replace the service installed on the host (Bellman). Not the client.

[https://hub.docker.com/r/jkarlos/git-server-docker/

This image] looks good. It is built on alpine and it has support for keys and data in volumes.

Should drop right in on Bellman and replace the system level version.

Git LFS

This is limited in the free account to 1GB of storage and 1GB of bandwidth. These days that means you can store one file. :-)

Using YouTube is probably a better idea.

LFS = Large File Storage = A special place for large files like binaries and videos. The LFS storage can be on your own server, it's just a web server.

Basically you add files to the repository using a new tool, and from then on references to the files are maintained properly. (It puts a linking file into the regular repo.)

Command docs are here. https://github.com/git-lfs/git-lfs/tree/main/docs/man

git lfs install         just once on the computer; works in Windows too
git lfs track "*.mp4"   in the repo, tell it to track mp4 files
git lfs track "*.zip"
git add .gitattributes  track this file too

If you already have files in your archive you must migrate to LFS

git lfs migrate

==