Working with git
Most work I do is out on github in public repos that anyone can access, search for geo-ceg and brian32768. I can't put every bit of my life out on the Internet, so I still operate my own Git server for proprietary projects. I also work in other peoples' servers for projects.
This page contains notes both on Github and on using private Git servers.
Git commands that I use most often
The main ones, obviously: init, clone, add, commit, status, push, pull (instead of pull everyone SHOULD use fetch and merge)
The ones I forget about: log
Until today I never bothered with branches (missing half the fun!) because 99.9% of the time right now I work on code by myself. This is a stupid excuse since it means having a broken master branch on Github most of the time as I do nightly checkpoints to switch between home and work. So, read the section below on branches.
I should do more tagging.
Recovering from staging a file by accident:
git add oops.cs git reset HEAD oops.cs
"Master" -> "Main"
At this point I have some repos with "master" as the default branch, some with "main", and some with "release". Over time they will migrate to "main". Randomly. Yeesh.
2021-02 - It's possible to simply rename a master branch in github now. They opted for "main".
Changing "master" to "main"
Recently github added rename ability to make moving to "main" easier. You can still do it this way though. These instructions I found on Stack Overflow and modified to suit my own use.
git commit -m "Commit any pending changes." -a git push git branch main # create branch git checkout main # switch to the new branch git push -u origin main # push the branch to the remote and track it git branch -d master # delete local master
Go to the repository at https://github.com/ and bring up the Settings page. Go to Branches and make "main" the main branch.
git push --delete origin master # delete remote master git remote prune origin # delete the remote tracking branch
git checkout -b dev
Now get on with life, working and testing in "dev" and merging into "main".
Setting execute bit
After the file is already in the repo,
git update-index --chmod=+x path/to/file
On add (a better way IMO),
git add --chmod=+x path/to/file
Using update-index is awkward because you then have to change the file before you can commit the +x.
Github workflow with "main" and "dev" branches
I am (from now on) following "the Driessen model". Read this carefully, especially regarding feature branches, which I started using today. :-) Or maybe soon. It's well written and there is no point in reinventing it here. HOWEVER there is value for me in running through my own tests here so I can remember what I did!
All day-to-day work is done in the "dev" branch. After confirming everything works in "dev", changes are merged into "main" branch and I tag it as a release.
Keep in mind that Git is distributed so a branch can exist on your local machine and never touch the repository. The entire branch lifecycle can stay on your local hard drive if you don't need to share (or move between two computers the way I do every day). Alternatively, you can have a branch track a remote repository.
The excellent free Pro Git book on the Git site explains branching in more detail including getting rid of a branch and dealing with merge conflicts.
Here is another explanation about the nuts and bolts of merging. http://longair.net/blog/2009/04/16/git-fetch-and-merge/
Normal project workflow
If "dev" code not working, don't do merges to main branch, duh, it's for releases!
Let's try all this out.
I am using my personal github account "brian32768", you use your own. :-)
Create a new empty github project called "brian32768/gt". (I let it add README.md and LICENSE.txt) Before going further, make sure it's "main" and not "master", and create a "dev" branch, and if you need to, make "main" the default, and delete "master".
Create a local, working copy.
cd ~/source/repos git clone [email protected]:brian32768/gt.git cd gt
Create a development branch "dev".
git checkout -b dev
Show the branch we're on; there will be two now "dev" and "main". "git status" shows the branch, too.
Henceforth do all development and testing on "dev"
Edit the README.md
Add a new file.
vi index.html git add index.html
Stage the changes and push them up to github
git commit -m 'here we go a coding, ra ra ra' -a git push --set-upstream origin dev
(Just the usual "git push" is not good enough the first push, you have to use set-upstream to create the branch in the repo.)
If you look in github now you should see both "main" and "dev" branches. The "main" branch should be untouched at this time.
Work in another place
I should be able to go home, clone on Murre, move to the "dev" branch, and make changes there. (Or I could just chdir to a separate place on my work computer, to mimic this workflow. :-)
Assuming that I've created a dev branch as in the previous section,
git clone [email protected]:brian32768/gt.git gt_dev2 cd gt_dev2 git checkout -b dev origin/dev
Leave off "origin/dev" and it will create a new local branch called "dev", and this can be confusing! You can do this when you are on the local dev branch, and this will push the code to github and create a new branch there at the same time.
git push --set-upstream origin dev
Now I can see my typo when I look at README.md so I know I am in the right place.
vi README.md git commit -m "made a change at home" git push
Now when I go back to computer #1 I need to get any changes I made at home, since both working directories (at work and at home) are on the "dev" branch, this should work.
cd source/repos/gt git fetch
There should not be any changes on the branch to merge so no 'git merge' needed here.
Now check contents of README.md to make sure the change came in correctly.
Merge dev changes into main
When you think you have working code, you must merge dev changes back into main.
Assuming you are currently on the dev branch, stage any changes and push them.
git commit -a -m 'good to go' git push
Switch to the main branch
git checkout main
Pull changes from dev branch into main
git merge dev
Jump back to the dev branch and continue working.
git checkout dev
Revert a change
For the purposes of my test, let's say that the recent change was wrong and I want to revert main to the previous version. Find the commit; if you tagged this is the same as using the tag.
git log commit a5b7b10caac48a5514ce781c28b71846ff8d1e5b (HEAD -> dev, tag: 1.0, origin/dev) Author: Brian Wilson <[email protected]> Date: Wed Nov 7 16:24:10 2018 -0800 added logo
commit d7a26f4c868bf29c749600f62386521d35a3044b Author: Brian Wilson <[email protected]> Date: Wed Nov 7 16:04:12 2018 -0800 typo fixed
(Ugh, did not mean to add the logo. Previous commit was "d7a26f4c868bf29c749600f62386521d35a3044b")
Now I am back at the previous version in main. I can leave things this way indefinitely while I deal with the issue in "dev", or I can do the same thing in "dev" - revert to the previous version to get rid of the broken changes. In this demo I just push ahead.
So I fix "dev". Then, I check in the changes on "dev" and merge to "main" again. This time the change is fine.
"Dev" is just the normal day to day stuff, and when I create a feature branch it's off of "dev". When I create a branch to support a specific production version that uses different assets (for example special branding), that's branched directly off "main".
Working with variants
The other use case I want to test is how to work with a variant branch; in this case, I want to create two different web sites with different branding.
To test this, I add the generic "map46.svg" and accompanying CSS style sheet to "main".
I create the variant branch and add a Clatsop County ""cclogo.gif" logo there with a different style sheet that will insert this graphic.
Now the question is, how do I support both? I make a change to "dev", merge it back to "main", and then I have to merge changes from "main" to variant without stepping on the assets. Since the assets did not change on "main", this should be no problem.
If I do change the (shared) style sheet for example, then I will have to do a more careful merge.
In a working directory
git tag -a 1.0 -m "First major release" git push origin 1.0
If you are doing a release now you can go up to the release/tags page on Github and mark this tag as a release.
To see what tags exist,
git tag -n
To see the datestamp,
git log --tags --simplify-by-decoration --pretty="format:%ci %d"
To check out this tagged version
git clone [email protected]:brian32768/gt git checkout -b version1 1.0
You've basically now checked out an outdated or variant branch so be careful about merging new code and committing, you'd have to move the tag forward or preferably use a new tag such as 1.0.1.
I have an outdated fork and need to pull in changes from the original repo before I can send a new pull request.
git remote add upstream original repository git fetch upstream git checkout main git rebase upstream/main
Making some changes
git checkout -b env_doc_edit edit the file(s) commit changes
Migrating git projects to Github.com
I have migrated most of my shareable projects to github.com to get them off my own server, making them more accessible and creating an off-site backup at no cost to me. Pretty much all the same git commands below work.
Migration: Use the Github Importer.
I used the command line import method. Basically you pull the repo with all its history onto your local machine then push it up to github. In brief
- Create the repository first at github, "ssurgo" in this case.
- Make a clone of your existing repo: git clone --bare [email protected]:ssurgo.git
- cd ssurgo
- Push everything up to github: git push --mirror https://github.com/geo-ceg/ssurgo.git
- cd ..
- Make sure it worked by cloning it! git clone [email protected]:Geo-CEG/ssurgo.git
- If it worked you have a directory "ssurgo" now. Remove the bare repo: rm -rf ssurgo.git
- Try commiting by making a README. emacs README.me
- git add README.md
- git commit -a -m "added README!"
- git push
Add Github repositories from the command line
conda create hub conda activate hub conda install hub for i in `cat folders`; do cd $i git init -b main git add . git commit -m initial\ commit -a hub create -d "A widget from Esri \"Web AppBuilder, Developer Edition\"." Wildsong/wabde-widget-$i.git git remote -v git push origin HEAD
for i in `cat folders`; do git tag touch README.md echo "= $i" >> README.md cat ../README.md >> README.md cp ../LICENSE . git add README.md LICENSE git commit -m 'added generic README and LICENSE files.' git push
Use case: you moved a repo from a personal account to an organization account
You can do that on github.com but now you need to change the repo, actually you don't have to but it's tidier.*
git remote set-url origin newurl
- If you don't change it, github leaves a link behind to redirect things and it will all be fine.
Use case: git for web app development
One of the main reasons I want my own git server is so that I can sync a web app between my laptop and a server.
- Develop and test test test on the laptop, in the "dev" branch (or in a "feature" branch").
- When you think it's flawless, push changes to the git server, still on the "dev" branch.
- On the web server, move onto the dev branch and pull the changes down.
- Test live server. If the live server breaks after an update, switch back to "main" and pull to revert.
- After confirming web site is working, merge changes into "main", tag, and push.
- On web server, pull "main" down to update it.
With github I use SSH pretty much exclusively and setting up keys is documented.
At work I have to use a standalone git server that uses Active Directory.
On Windows, I install the git-scm ("Git for Windows") package and use its default git-credential authentication code. https://git-scm.com/docs/git-credential
I set up a file called "cred" and fed it to git, the file contains
protocol=https host=MYGITHOSTNAME like codebase.domain.com username=MYNAME password=MYPASSWORD
then I did
git credential approve < cred
I then tested a clone operation and it did not prompt for credentials so I am happy. Well, except I don't know where it keeps the creds on Windows. I moved "cred" and tried again and the command still works.
On Debian, first I told it to use a global settings file (~/.git-credentials) by typing
git config --global credential.helper store
I used a git clone command and it stored the credentials after I successfully connected one time. Note they are stored in that file in plain text. Caveat Emptor.
Running a git server
File ownerships and ssh access control
Access to the server is through ssh. The repositories will be owned by the user 'git'. So I put the home for user 'git' where I want the repositories to be; on Bellman I use /green/repositories. Then I set up ssh with keys so that I don't need to share passwords.
I put create a separate 'git' key pair and put the public key into ~git/.ssh/authorized_keys I can test it with "ssh [email protected] list", this should list the available repos and exit.
Once that's all in place I don't need a complete path for the "remote" command, just one relative to git's home.
Here is how I generate a key for github
ssh-keygen -t rsa -b 4096 -C my_email_address eval $(ssh-agent -s) ssh-add ~/.ssh/id_rsa_github
Here is a quick way to test it
ssh [email protected] . . . Hi brian32768! You've successfully authenticated, but GitHub does not provide shell access.
Creating a new private repo
I am using ssh for file transfer.
I find it easiest to create a new empty repo directly on the server and then push existing content into it.
ssh bellman cd ~git sudo git init --bare myproject.git create a bare repository sudo chown -R git.wildsong myproject.git exit cd source/repos assuming myproject is a folder containing the files you want in a repo cd myproject git init make a gitignore file in myproject git add some files git commit -m 'initial commit' git remote add origin [email protected]:myproject.git git push origin main this pushes committed files from the local repository to the "main" branch on the remote machine
Now I should be able to clone my project onto the web server and start using git to keep it updated.
ssh webserver cd /var/www/appserver git clone git.wildsong.biz:myproject.git
On my copy I need to define the remote repo,
git remote add origin [email protected]:myproject.git
When I make changes on the local laptop, first I commit them locally.
..then I push them up to the git server
git push origin main
..then I pull them down onto the public web server
Then I create a virtualenv and load the requirements.
cd myproject virtualenv env source env/bin/activate pip install -r requirements.txt python run.py
Now I want to contribute to someone's project. This description is good especially after reading the comments section too.
Briefly, leaving out some critical steps...
- Fork -- Makes a copy of the project in Github that I can change
- Clone -- Pull the code down from Github to my computer
- Branch -- I will work in a branch and then remove it when done
- Edit -- Make proposed changes
- Issue pull request -- Notifies project owner; hopefully it will be accepted
I keep some code in github, and I want to incorporate it into a project. I can use the submodule feature.
cd myproject git submodule add [email protected]:brian32768/pyst2.git
This clones pyst2 into a subdirectory (called pyst2 by default) Now I can work on pyst2 and do commits and pushes but it remains separate from 'myproject'. If I ask for status...
git status On branch "main" Your branch is up-to-date with 'origin/main'. Changes to be committed: (use "git reset HEAD <file>..." to unstage)
modified: .gitmodules new file: pyst2
When you clone a new copy or update an existing clone (git pull) then the submodules will be there but empty, you have do do this
git submodule update --init
That will tell git to populate each submodule folder with code from the other repository.
I am not sure why but it left the HEAD detached so I had to do
git checkout main
Git Server in Docker
This would be to replace the service installed on the host (Bellman). Not the client.
This image] looks good. It is built on alpine and it has support for keys and data in volumes.
Should drop right in on Bellman and replace the system level version.
This is limited in the free account to 1GB of storage and 1GB of bandwidth. These days that means you can store one file. :-)
Using YouTube is probably a better idea.
LFS = Large File Storage = A special place for large files like binaries and videos. The LFS storage can be on your own server, it's just a web server.
Basically you add files to the repository using a new tool, and from then on references to the files are maintained properly. (It puts a linking file into the regular repo.)
Command docs are here. https://github.com/git-lfs/git-lfs/tree/main/docs/man
git lfs install just once on the computer; works in Windows too git lfs track "*.mp4" in the repo, tell it to track mp4 files git lfs track "*.zip" git add .gitattributes track this file too
If you already have files in your archive you must migrate to LFS
git lfs migrate