Gitlab CI runner installation

The instructions to install GitLab CI runner are adapted to Ubuntu 14.04 to connect to GitLab CI and run jobs when a commit is pushed to a branch.

A runner token must be obtained from GitLab CI, at the http://cong.dachary.org:8080/projects/1/runners URL for instance.

The gitlab-ci-multi-runner/ is installed as follows:

$ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash
$ sudo apt-get install gitlab-ci-multi-runner
$ $ sudo gitlab-ci-multi-runner register
Please enter the gitlab-ci coordinator URL (e.g. http://gitlab-ci.org:3000/):
http://cong.dachary.org:8080/
Please enter the gitlab-ci token for this runner:
441877520384923424
Please enter the gitlab-ci description for this runner:
[cong]: runner1
INFO[0156] 4418775e Registering runner... succeeded
Please enter the executor: shell, parallels, docker, docker-ssh, ssh:
[shell]: docker
Please enter the Docker image (eg. ruby:2.1):
golang
If you want to enable mysql please enter version (X.Y) or enter latest?

If you want to enable postgres please enter version (X.Y) or enter latest?

If you want to enable redis please enter version (X.Y) or enter latest?

If you want to enable mongo please enter version (X.Y) or enter latest?

INFO[0281] Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

It is configured to run each job in a golang docker container. The project git repository is expected to have a .gitlab-ci.yml file at the root. For instance if .gitlab-ci.yml was:

job1:
  script: "type go"

the GitLab runner would succeed with:

Mirror github pull requests locally

Each GitHub pull request is associated with a reference in the target repository. For instance the commit sent to pull request 3948 is the reference refs/pull/3948/head. If GitHub successfully merges the pull request in the target branch, another reference is associated with it, for instance refs/pull/3948/merge.
It is convenient to mirror pull requests in local branches, for instance to trigger GitLab CI or jenkins git-plugin because they only react to commits that belong to branches, not orphaned references. The tip of the branch can be named pull/XXX and its tip reset to the matching merge reference.
The tip of the branch could be set to the head reference but it would be less effective than using the merge because it is based on an older version of the target branch. Whatever test runs on the head, it may fail although it could succeed after the merge.
GitHub does not set the merge and the head reference atomically: the head can be set before GitHub even tries to merge. Similarly, when a pull request is rebased and forced push, there is a window of opportunity for the merge reference to still be about the previous head.
The following shell function takes care of all these border cases and keeps an up to date set of branches accurately reflecting all pull requests from a GitHub repository:

function import_pull_requests() {
    local remote=$1

    local ref
    local remote_head

    git fetch $remote +refs/pull/*:refs/remotes/$remote/pull/*

    git for-each-ref \
        --sort='-committerdate' \
        --format='%(refname) %(objectname)' \
        refs/remotes/$remote/pull/*/head | \
        while read ref remote_head ; do

        local pr=$(echo $ref | perl -pe 's:.*/pull/(.*)/head:$1:')

        # ignore pull requests that cannot merge
        local merge=$(git rev-parse --quiet --verify
            refs/remotes/$remote/pull/$pr/merge)
        test -z "$merge" && continue

        # ignore pull requests for which the merge does not match the
        # remote head, most likely because it has not been updated yet
        # after a rebase was pushed
        local merged_head=$(git rev-parse
            refs/remotes/$remote/pull/$pr/merge^2)
        test "$merged_head" != "$remote_head" && continue

        # nothing to do if the head did not change since we last saw
        # it
        local local_head=$(git rev-parse --quiet --verify
            refs/pull/$pr/head)
        test "$remote_head" = "$local_head" && continue

        # remember the head for the next round
        git update-ref refs/pull/$pr/head $remote_head

        # create/update a branch with the successfull merge of the
        # head
        git update-ref refs/heads/pull/$pr $merge
        echo branch pull/$pr
    done
}

Download the function and the associated tests

Customizing the gitlab home page

The customization of the Gitlab home page is a proprietary extension that is not available in the Free Software version. When running Gitlab from docker containers, the home page template needs to be moved to a file that won’t go away with the container:

$ layouts=/home/git/gitlab/app/views/layouts/
$ docker exec gitlab mkdir -p /home/git/data/$layouts
$ docker exec gitlab mv $layouts/devise.html.haml /home/git/data/$layouts
$ docker exec gitlab ln -s /home/git/data/$layouts/devise.html.haml \
   $layouts/devise.html.haml

The template can now be modifed in /opt/gitlab/data/home/git/gitlab/app/views/layouts/ from the host running the container. It is a HAML template which can have raw HTML as long as proper indentation is respected.

Continue reading “Customizing the gitlab home page”

Gitlab CI runner installation

This content is obsolete

The instructions to install GitLab CI runner are adapted to Ubuntu 14.04 to connect to GitLab CI and run jobs when a commit is pushed to a branch. The recommended packages are installed except postfix and with ruby2.0 and ruby2.0-dev in addition:

sudo apt-get update -y
sudo apt-get install -y wget curl gcc libxml2-dev libxslt-dev \
   libcurl4-openssl-dev libreadline6-dev libc6-dev \
   libssl-dev make build-essential zlib1g-dev openssh-server \
   git-core libyaml-dev libpq-dev libicu-dev \
   ruby2.0 ruby2.0-dev

Ruby2.0 is made the default ruby interpreter

sudo rm /usr/bin/ruby /usr/bin/gem /usr/bin/irb /usr/bin/rdoc /usr/bin/erb
sudo ln -s /usr/bin/ruby2.0 /usr/bin/ruby
sudo ln -s /usr/bin/gem2.0 /usr/bin/gem
sudo ln -s /usr/bin/irb2.0 /usr/bin/irb
sudo ln -s /usr/bin/rdoc2.0 /usr/bin/rdoc
sudo ln -s /usr/bin/erb2.0 /usr/bin/erb
sudo gem update --system
sudo gem pristine --all

The bundler gem is installed

sudo gem install bundler

and the GitLab CI runner user created

sudo adduser --disabled-login --gecos 'GitLab CI Runner' gitlab_ci_runner

The GitLab CI runner code is installed in the home of the corresponding user with:

sudo su gitlab_ci_runner
cd ~/
git clone https://gitlab.com/gitlab-org/gitlab-ci-runner.git
cd gitlab-ci-runner
bundle install --deployment

The CI token is retrieved from the GitLab CI pannel

and used to grant access to the runner:

CI_SERVER_URL=http://workbench.dachary.org:8080 \
  REGISTRATION_TOKEN=778b1d4856f26da392a bundle exec ./bin/setup

The daemon is started from root with:

su gitlab_ci_runner -c 'cd $HOME/gitlab-ci-runner ; bundle exec ./bin/runner'

The GitLab CI interface shows the runner as registered:

Assuming all the above was done from within a docker container, it can be persisted as an image with

docker commit b504ab6ba122 gitlab-runner

and used to multiply the runners with:

$ docker run --rm -t gitlab-runner \
  su gitlab_ci_runner -c  'cd $HOME/gitlab-ci-runner ; \
  CI_SERVER_URL=http://workbench.dachary.org:8080 \
  REGISTRATION_TOKEN=b14852619da392a \
  bundle exec ./bin/setup ; bundle exec ./bin/runner'
Registering runner with registration token: 2619da3, url: http://workbench.dachary.org:8080.
Runner token: 35f9d40f2e072487870f987
Runner registered successfully. Feel free to start it!
* Gitlab CI Runner started
* Waiting for builds
2014-12-06 17:18:26 +0000 | Checking for builds...nothing
2014-12-06 17:20:27 +0000 | Checking for builds...received
2014-12-06 17:20:27 +0000 | Starting new build 6...
2014-12-06 17:20:27 +0000 | Build 6 started.
2014-12-06 17:20:32 +0000 | Submitting build 6 to coordinator...ok
2014-12-06 17:20:33 +0000 | Completed build 6, success.
2014-12-06 17:20:33 +0000 | Submitting build 6 to coordinator...aborted
2014-12-06 17:20:38 +0000 | Checking for builds...nothing
...

When the container is stopped, the runner must be manually removed from the GitLab CI. Projects in the GitLab CI will be confused by the disapearance of the runner and must be removed and re-added otherwise no job will get scheduled.
It is easier to install on Fedora 20

sudo gem install bundler

sudo useradd -c 'GitLab CI Runner' gitlab_ci_runner
export PATH=/usr/local/bin:$PATH
cd ~/
git clone https://gitlab.com/gitlab-org/gitlab-ci-runner.git
cd gitlab-ci-runner
bundle install --deployment
CI_SERVER_URL=http://workbench.dachary.org:8080 \
  REGISTRATION_TOKEN=XXXXX bundle exec ./bin/setup

Gitlab CI installation

Assuming a GitLab container has been installed via Docker, a GitLab CI can be installed and associated with it. It needs a separate database server:

sudo mkdir -p /opt/mysql-ci/data
docker run --name=mysql-ci -d -e 'DB_NAME=gitlab_ci_production'  \
 -e 'DB_USER=gitlab_ci'  \
 -e 'DB_PASS=XXXXX'  \
 -v /opt/mysql-ci/data:/var/lib/mysql  sameersbn/mysql:latest

but it can re-use the redis server from GitLab.
Create a new application in GitLab (name it GitLab CI for instance but the name does not matter) and set the callback URL to http://workbench.dachary.org:8080/user_sessions/callback. It will display an application id to use for GITLAB_APP_ID below and a secret to use for GITLAB_APP_SECRET below.

docker pull sameersbn/gitlab-ci
sudo mkdir -p /opt/gitlab-ci/data
pw1=$(pwgen -Bsv1 64) ; echo $pw1
pw2=$(pwgen -Bsv1 64) ; echo $pw2
docker run --name='gitlab-ci' -it --rm   \
  --link mysql-ci:mysql \
  --link redis:redisio \
  -e 'GITLAB_URL=http://workbench.dachary.org'  \
  -e 'SMTP_ENABLED=true' \
  -e 'SMTP_USER=' \
  -e 'SMTP_HOST=172.17.42.1'  \
  -e 'SMTP_PORT=25'  \
  -e 'SMTP_STARTTLS=false'  \
  -e 'SMTP_OPENSSL_VERIFY_MODE=none'  \
  -e 'SMTP_AUTHENTICATION=:plain' \
  -e 'GITLAB_CI_PORT=8080'  \
  -e 'GITLAB_CI_HOST=workbench.dachary.org'  \
  -e "GITLAB_CI_SECRETS_DB_KEY_BASE=$pw1" \
  -e "GITLAB_CI_SECRETS_SESSION_KEY_BASE=$pw2" \
  -e "GITLAB_APP_ID=feafd' \
  -e 'GITLAB_APP_SECRET=ffa8b' \
  -p 8080:80  \
  -v /var/run/docker.sock:/run/docker.sock  \
  -v /opt/gitlab-ci/data:/home/gitlab_ci/data  \
  -v $(which docker):/bin/docker  sameersbn/gitlab-ci

It uses port 8080 because port 80 is already in use by GitLab. The SMTP* are the same as when GitLab was installed.

The user and password are the same as with the associated GitLab.

Copy a github pull request to gitlab

A mirror of a github repository is setup and contains two remotes:

gitlab	 git@workbench.dachary.org:tests/testrepo.git (push)
origin	 https://github.com/loic-bot/testrepo (push)

The github2gitlab command of gh (run from ~gitmirrors/repositories/Tests/testrepo) creates a merge request in gitlab by copying the designated pull request from github:

$ gh gg --user loic-bot --repo testrepo --number 3

Original github pull request

Matching gitlab merge request

Continue reading “Copy a github pull request to gitlab”

Gitlab workbench

Gitlab is installed on http://workbench.dachary.org using docker images. redis is installed first, as an independant container:

docker pull sameersbn/redis:latest
docker run --name=redis -d sameersbn/redis:latest

then MySQL

docker pull sameersbn/mysql:latest
docker run --name=mysql -d \
  -e 'DB_NAME=gitlabhq_production' \
  -e 'DB_USER=gitlab' \
  -e 'DB_PASS=XXXXXXXXXXXX' \
  -v /opt/mysql/data:/var/lib/mysql \
  sameersbn/mysql:latest

and finally gitlab

docker pull sameersbn/gitlab:latest
docker run --name='gitlab' -it -d  \
  --link mysql:mysql --link redis:redisio \
  -e 'GITLAB_EMAIL=gitlab@workbench.dachary.org'  \
  -e 'SMTP_ENABLED=true' \
  -e 'SMTP_DOMAIN=workbench.dachary.org' \
  -e 'SMTP_USER=' \
  -e 'SMTP_HOST=172.17.42.1' \
  -e 'SMTP_PORT=25' \
  -e 'SMTP_STARTTLS=false' \
  -e 'SMTP_OPENSSL_VERIFY_MODE=none' \
  -e 'SMTP_AUTHENTICATION=:plain' \
  -e 'GITLAB_SIGNUP=true' \
  -e 'GITLAB_PORT=80' \
  -e 'GITLAB_HOST=workbench.dachary.org' \
  -e 'OAUTH_ALLOW_SSO=true' \
  -e 'OAUTH_BLOCK_AUTO_CREATED_USERS=false' \
  -e 'OAUTH_GITHUB_API_KEY=github Client ID'  \
  -e 'OAUTH_GITHUB_APP_SECRET=github Client Secret' \
  -e 'GITLAB_SSH_PORT=22' \
  -p 22:22 -p 80:80 \
  -v /var/run/docker.sock:/run/docker.sock \
  -v /opt/gitlab/data:/home/git/data \
  -v $(which docker):/bin/docker \
  sameersbn/gitlab

The ssh server of the server will need to bind another port by editing /etc/ssh/sshd_config, changing the Port value and restarting the server with stop ssh ; start ssh.
The OmniAuth single sign on is configured following gitlab instructions, except for editing the config.yml file: the OAUTH_GITHUB_* are set instead, using information found in the applications settings github page.
It uses the automagic dockerlinks to connect it to the MySQL and redis servers (–link mysql:mysql –link redis:redisio). The SMTP server is configured using variables from the documentation to point to the server running on the host (172.17.42.1 is the IP of the docker0 bridge on which all containers are connected and in the same IP range as the dynamic IP they are given). A postfix server is installed on the host:

$ sudo apt-get install postfix
... chose internet server ...

and it is configured to accept to relay mails from any docker contain in the 172.0.0.0/8 IP range:

$ cat /etc/postfix/main.cf
...
myhostname = workbench.dachary.org
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = workbench.dachary.org, localhost, localhost.localdomain, localhost
relayhost =
mynetworks = 172.0.0.0/8 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
...

A working SMTP server is required to allow sign up as required with GITLAB_SIGNUP=true. The gitlab persistent data is in /opt/mysql/data (bind mounted with -v /opt/mysql/data:/var/lib/mysql) for the MySQL database and /opt/gitlab/data (bind mounted with -v /opt/gitlab/data:/home/git/data) for repositories, gitlab assets etc. When the host reboots, the containers can be restarted as above, they only contains non persistent information.