Simple Gitlab CE HA solution

Have you ever wondered if it's possible to run Gitlab CE in a HA environment? Simple answer is yes and most important you don't have to pay a dime for this.

Gitlab CE HA explained

Long story short for a basic Front-end HA setup you're going to need at least 3 servers. This article won't get into the details of setting up database replication and load-balancing it.

Requirements

Three servers running any Gitlab CE compatible OS, I'm using CentOS 7.x.

  • node-gitlab01.invalid - 10.8.0.10
  • node-gitlab02.invalid - 10.8.0.11
  • node-pgsql01.invalid - 10.8.0.12 and 10.8.0.13
  • A node to run NFS server and HAproxy

For simplicity you can reuse the DB node.

Installation steps

PostgreSQL Database

First of all begin with the DB node. Install PostgreSQL 9.2 on the server and create 2 database instances, one for Gitlab, one for Gitlab CI and assign users to them. For the purpose of this tutorial I'm using gitlab-prod and gitlab-prod-ci. Gitlab requires Redis instance as-well so install it. With this our initial work on that node finished. We're going to come back to it later on in this article.

First front-end node

Download and install Gitlab CE's RPM from their repository. NOTE: Don't run gitlab-ctl reconfigure as it installs PostgreSQL and Redis on the same node by default. Instead we want to point them to our already installed PostgreSQL and Redis server. The following example is the bare minimum you need to proceed. Don't forget to replace db_username and db_password with the ones you're selected while creating the DB server.

/etc/gitlab/gitlab.rb

external_url 'http://gitlab-ha.invalid'  
gitlab_rails['db_adapter'] = "postgresql"  
gitlab_rails['db_encoding'] = "unicode"  
gitlab_rails['db_database'] = "gitlab-prod"  
gitlab_rails['db_pool'] = 10  
gitlab_rails['db_username'] = "gitlab"  
gitlab_rails['db_password'] = "somefancyandhardtoguesspass"  
gitlab_rails['db_host'] = "node-pgsql01.invalid"  
gitlab_rails['db_port'] = 5432  
gitlab_rails['redis_host'] = "node-pgsql01.invalid"  
gitlab_rails['redis_port'] = 6379  
gitlab_rails['redis_database'] = 0  
postgresql['enable'] = false  
redis['enable'] = false  
nginx['enable'] = true  
nginx['redirect_http_to_https'] = true  
gitlab_ci['db_adapter'] = "postgresql"  
gitlab_ci['db_encoding'] = "unicode"  
gitlab_ci['db_database'] = "gitlab-ci-prod"  
gitlab_ci['db_pool'] = 10  
gitlab_ci['db_username'] = "gitlab_ci"  
gitlab_ci['db_password'] = "somefancyandhardtoguesspass2"  
gitlab_ci['db_host'] = "node-pgsql01.invalid"  
gitlab_ci['db_port'] = 5432  
gitlab_ci['redis_host'] = "node-pgsql01.invalid"  
gitlab_ci['redis_port'] = 6379  
ci_redis['enable'] = false  

Once you've set your gitlab.rb properly run

Initial Gitlab Server configuration:

gitlab-ctl reconfigure  

Gitlab will do the magic and install and configure all necessary components skipping PostgreSQL and Redis local instances. At this point your DB is initialized. You can test proper functionallity by accessing http://node-gitlab01.invalid. We can move now to the next step.

Second front-end server

Again install Gitlab CE from their official repository skipping gitlab-ctl reconfigure step. Then copy gitlab.rb and gitlab-secrets.json from your first server. The important bit is your gitlab-secrets.json file that holds all encryption keys to access DB data.

Run Gitlab server configuration:

gitlab-ctl reconfigure  

again. It'll just configure the neccessary components, skipping DB initialization. Again test proper functionality by accessing http://node-gitlab02.invalid.

HAproxy and NFS configuration

Login again on the node-pgsql01 server and install HAproxy and NFS server

HAproxy and NFS installation:

yum install haproxy nfs-utils  

Create a folder and NFS export it with suitable permissions

/etc/exports

/exports/.ssh                   *(rw,sync,no_root_squash,no_all_squash)
/exports/git-data               *(rw,sync,no_root_squash,no_all_squash)

Now login to both node-gitlab servers, stop Gitlab and mount these shared folders.

Stop Gitlab:

gitlab-ctl stop  

Edit fstab

10.8.0.12:/exports/.ssh        /var/opt/gitlab/.ssh        nfs rw,noauto,user  0 0  
10.8.0.12:/exports/git-data    /var/opt/gitlab/git-data    nfs rw,noauto,user  0 0  

Mount the folders

mount /var/opt/gitlab/.ssh  
mount /var/opt/gitlab/git-data  

Don't forget to start Gitlab on both nodes

gitlab-ctl start  

With this being done get back to node-pgsql01 and configure HAproxy. I'm using the following configuration.

/etc/haproxy/haproxy.cfg

global  
    daemon
    maxconn    30000
    user     haproxy
    group     haproxy
    chroot      /var/lib/haproxy
    log     /dev/log    local0
    log     /dev/log    local1 notice

defaults  
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend http-in  
    bind 10.8.0.13:80
    default_backend http-servers

frontend ssh-in  
    mode tcp
    #option tcplog
    bind 10.8.0.13:22
    default_backend ssh-servers

backend ssh-servers  
    mode tcp
    option allbackups
    timeout server 2h
    server node-gitlab01 10.8.0.10:22
    server node-gitlab02 10.8.0.11:22 backup

backend http-servers  
    option allbackups
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:node-pgsql01.invalid
    server node-gitlab01 10.8.0.10:80 check maxconn 5000
    server node-gitlab02 10.8.0.11:80 check maxconn 5000 backup

listen admin  
    bind 10.8.0.13:8080
    stats enable
    stats uri /
    stats hide-version
    stats auth admin:secret

Please note that I'm using the alias IP for accessing Gitlab. You probably already realized that gitlab-ha points to it as it's configured as external URL for my Gitlab servers. One final change is required before you attempt to start HAproxy. Edit /etc/ssh/sshd_config and set your listen address to the primary IP of node-pgsql01 else the port will collide with the one in HAproxy. Start HAproxy and try to login to Gitlab CE at http://gitlab-ha.invalid

comments powered by Disqus