SVN to GitHub Migration

Repository Migration

This HowTo reflects the steps needed for converting a SVN repository into a Git repo and pushing it to GitHub. The steps have been tested with a few projects, so apply at own risk. Your SVN repository will stay untouched, so you will always have a backup as SVN.

The filesystem commands are for Unix shell, Git for Windows comes with the "Git Bash" which should be capable of executing all commands.

Git with Git-SVN is required for the following actions. For details about the used commands take a look at the git svn documentation.

Initial clone

A few things need to be setup to create a local Git repository from a Remote SVN location. First, you need to checkout the complete SVN repo (the base, having trunk/tags/branches as subfolders) to your local filesystem. This is needed to determine all committers from the complete history. Navigate to your home folder and create a working directory, e.g. "svn-to-git-work", and navigate into it. |

cd ~
mkdir svn-to-git-work
cd svn-to-git-work

|

Checkout the base SVN repository (i.e. the directoy that contains the directories trunk, branches, and tags) |

svn co https://svn.52north.org/svn/swe/myproject/ myproject_svn
# checkout of the SES base repo into the folder "myproject_svn"

|

Authors file

As we have a local copy of the SVN repo, we are able to find all committers: |

cd myproject_svn
svn log -q | grep -e '^r' | awk 'BEGIN { FS = "|" } ; { print $2 }' | sort | uniq
cd ..

|

The output provides a line-separated list of committing authors. These now need to be mapped to Git/!GitHub-Accounts and Emails. Important: Both username and email are required for GitHub accounts! Since you cannot find out the correct emails on GitHub yourself, this is a good opportunity to write an email to all contributors informing them that you want to move the code to GitHub and asking for their account details (username and email). Altnatively, you can use the known author mappings at SvnToGitAuthors (Restricted access! Only 52°North staff members!).

As committers in the 52n SVN repos are LDAP-managed, here is a base authors file. Feel free to add authors. The syntax is the following (the placeholders are wrapped in "( )" characters):

(SVN-Authorname) = (GitHub-Username) <(GitHub-Email)>

The email is mandatory. So and example line of such a file would look like this:

LucyDog = 52n-Lucy-Git <lucy@52north.org>

Some of the authors are former 52°North staff/contributors which probably do not have a GitHub account. GitHub does not complain about unavailable users, so we can just create these users ourselves. The default abbreviated naming can be applied ( FLA-52n for "Firstname Lastname", so for example XYZ-52n <info@52north.org>). Commits will then simply not be linked to a users GitHub page.

Store contents in a file called 52n-contributors.txt in your previously created work directory.

Git Svn Execution

Now we can start the initial clone. This will take a while depending on the size of the repository. Here we need the URL of your online SVN repository again, the last section (after the last space) is the name of the new local GitHub repository.

Go to the working directory (where the contributors file is located).

git svn clone --prefix=svn/ -s --no-metadata --authors-file=52n-contributors.txt https://svn.52north.org/svn/swe/myproject/ myproject_git

The above commands assumes the default SVN layout (trunk/tags/branches) by using the option "-s" and will complain if it finds an SVN commit with an author not defined in the authors file.

If you have not connected to the 52°North SVN before you might get a error about validating the certificate. You can check the fingerprint as described here here.

If the SVN is read protected you will be asked for a password. If the username is not correct (the machine username is used as default), just press enter and you will be asked for SVN username and password. You might have to authenticate yourself a couple of times before it works, no worries.

Only retaining a certain range of revision is also possible using the -r parameter and the syntax "startRevision:endRevision":

git svn clone -r 12345:HEAD --prefix=svn/ -s --no-metadata --authors-file=52n-contributors.txt https://svn.52north.org/svn/swe/myproject/ myproject_git

You can also manually define the trunk (to only migrate the trunk) using the --trunk parameter or define the locations of trunk, branches and tags manually. Please consult the git svn documentation. Example:

git svn clone -r 11367:HEAD --prefix=svn/ --no-metadata --authors-file=52n-contributors.txt --trunk=https://svn.52north.org/svn/myproject/ myproject_git

Make a Backup

At this point it is a good idea to create a backup of the initially cloned repo. If you make any mistakes in the following steps you can just restore the state with that backup:

cp -rf myproject_git myproject_git_bak

Applying Git Logic

As we now have our initial clone we need to do some manual adjustments.

Tags

Git uses Annotated Tags for labelling a specific commit object. Git-SVN does not do this automatically, so we need a script for that (source: http://blogs.atlassian.com/2012/01/moving-confluence-from-subversion-to-git/).

Save the following script to a file (e.g. "create_git_tags.sh") in the root of your newly created local Git repository (e.g. myproject_git), go to that directoy on the command line and execute with the command ./create_git_tags.sh .

#!/bin/sh # Based on https://github.com/haarg/convert-git-dbic set -u set -e git for-each-ref --format='%(refname)' refs/remotes/svn/tags/* | while read r; do tag=${r#refs/remotes/svn/tags/} sha1=$(git rev-parse "$r") commiterName="$(git show -s --pretty='format:%an' "$r")" commiterEmail="$(git show -s --pretty='format:%ae' "$r")" commitDate="$(git show -s --pretty='format:%ad' "$r")" # Print the commit subject and body separated by a newline git show -s --pretty='format:%s%n%n%b' "$r" | \ env GIT_COMMITTER_EMAIL="$commiterEmail" GIT_COMMITTER_DATE="$commitDate" GIT_COMMITTER_NAME="$commiterName" \ git tag -a -m "Tag: ${tag} sha1: ${sha1} using '${commiterName}', '${commiterEmail}' on '${commitDate}'" "$tag" "$sha1" # Remove the svn/tags/* ref git update-ref -d "$r" done

You can check the created tags with the command git tag -l, which lists all tags.

Don't forget to remove the .sh-file from the folder.

Branches

The SVN to Git conversion creates branches automatically. You can take a look at the branches:

$user@machine:~/git_work/SES$ git branch -a remotes/svn/SWC_Refactor * master

All branches with prefix /remotes/svn are converted SVN branches. To have them as Git branches in your repo, you must switch/checkout each of them once:

#this switches to the remote branch git checkout remotes/svn/SWC_Refactor #this creates a local branch from the remote (as it is the active one) git checkout -b SWC_Refactor

To continue the work, switch back to the master branch. In the final push, every checked-out branch will be included.

.gitignore

After the initial clone of the repo, you should add a file called ".gitignore" to the base directory. A common .gitignore (for Java/Eclipse projects) could look like:

target .classpath .project .settings

You can do this in any text editor or right on the command line with vi .gitignore, then add the lines above and save and close vi ( :wq).

After editing .gitignore it must be commited to the local repository: |

git add .gitignore
git commit -m "Adjust .gitignore"

|

Removing Unnecessary Branches, Tags

Removing Branches from the local Git repo should be safe, as SVN trunk and branches are decoupled. So, even if a SVN branch is merged back into the trunk, it is not linked to the branch in any way ( right?!).

tbd!

Removing Big Files

See SvnToGitHubMigrationBigFiles.

Push to GitHub

First, you need to create the repository on GitHub (e.g. for the 52°North organization account or your own account). It must be empty (no default README). Then you can push the local repository. In the process you are prompted for you github account and password.

# assuming you are in the directory of your cloned repo (e.g. "SES", or "SES-bare.git" if you removed unwanted files). # add a link to the remote repository git remote add origin https://github.com/52North/SES.git # Push all refs under refs/heads git push origin --all # Push all refs under refs/tags git push origin --tags

Congrats! Your code is on GitHub now.

Cleaning Up and Opening Up

Make SVN read-only

To avoid problems with multiple developers working on a project write to IT-support to make the now old SVN repository read-only.

Update Ohloh Project Page

All ("stable" or public) 52°North projects have a project page on Ohloh: https://www.ohloh.net/orgs/n52. Please update the code location to the new GitHub repo, or if the project does not yet have one, create a project!

Edit the tagline of the project

On top of the repository there is a tagline, a one/two sentence description of the project. Next to it is an "Edit"-link, and this is the right time to add a concise description of the project!

Add a README.md

The README.md is an important file, because it is the starting point for people to get to know the project and the repository. Add at least information on what the software does and links to more documentation (e.g. the 52°North wiki, a GitHub wiki, 52°North website). Other commonly added information are required software, installation guides, links to demo pages, latest build status, ...

The readme file is written in GitHub Flavoured Markdown: https://help.github.com/articles/github-flavored-markdown

Troubleshooting

Use of uninitialized value in substitution (s///) at [...]

Use of uninitialized value in substitution (s///) at C:\Program Files (x86)\Git/libexec/git-core\git-svn line 2097

See the solution in https://groups.google.com/forum/?fromgroups=#!topic/msysgit/7MQVwRO-2N4 - it worked for me. -- DanielNuest - 2013-06-14

Delta source ended unexpectedly

Error: git svn clone Incomplete data: Delta source ended unexpectedly at /usr/lib/perl5/site_perl/Git /SVN/Ra.pm line 282

Adding --ignore-paths="/branches/" to the git svn clone command solved this for me - based on some googling this might also be fixed by checking out a previous SVN revision. -- DanielNuest - 2014-02-18

Further Notes

• Optional improvement: Using a local SVN mirror. The initial clone can be accelerated with a local SVN mirror. Documentation tbd: svnadmin + svnsync