Production Architecture

Our core infrastructure is currently hosted on several cloud providers, all with different functions. This document does not cover servers that are not integral to the public facing operations of GitLab.com.

On this page

Diagram of the Architecture

Source, GitLab internal use only

Infrastructure "Services" and Their SLx's

In order for us to reach the goals around availability and latency for GitLab.com, we started by setting a target internal SLA for the service as a whole from the user's perspective. From those targets, we can work backwards through the architecture to determine what the Service Level objectives should be for the infrastructure "services" that support GitLab.com.

Since we are relying on hardware that itself only offers an SLA of 99.9% availability, we face an "SLA inversion" (read more about this here or here). For example, in the current situation, each time an NFS server goes down, this results in an outage of GitLab.com. Since we are only guaranteed 99.9% uptime per NFS server, the maximum SLA for GitLab.com as a whole will be <= (99.9%)N, where N is the number of NFS servers. To overcome this, the service that is offered by the NFS servers either needs to be redesigned in some way (e.g. through using Gitaly), or the application that depends on it needs to have a way to not go down when the NFS service is unavailable (i.e. graceful degradation). Similar considerations apply to things such as the cache, background jobs processing, availability of the database, and so on.

To tackle this challenge, we consider the following elements of the infrastructure to be "services" that should be able to meet their own internal SLAs:

Internal Networking Scheme

A visualization of the whole address space can be found here (GitLab internal use only).

Production

Virtual Network Name: GitLabProd

Resource Group: GitLabProd

IP space: 10.64.0.0/11

Subnet Name Subnet Range Tier Domain
ExternalLBProd 10.65.1.0/24 Load balancers lb.prd.gitlab.com
InternalLBProd 10.65.2.0/24 Load balancers lb.prd.gitlab.com
DBProd 10.66.1.0/24 Databases db.prd.gitlab.com
RedisProd 10.66.2.0/24 Databases db.prd.gitlab.com
ElasticSearchProd 10.66.3.0/24 Databases db.prd.gitlab.com
ConsulProd 10.67.1.0/24 Support Services infra.prd.gitlab.net
VaultProd 10.67.2.0/24 Support Services infra.prd.gitlab.net
DeployProd 10.67.3.0/24 Support Services infra.prd.gitlab.net
LogProd 10.68.1.0/24 Logging log.prd.gitlab.net
APIProd 10.69.2.0/23 Services sv.prd.gitlab.com
GitProd 10.69.4.0/23 Services sv.prd.gitlab.com
SidekiqProd 10.69.6.0/23 Services sv.prd.gitlab.com
WebProd 10.69.8.0/23 Services sv.prd.gitlab.com
RegistryProd 10.69.10.0/23 Services sv.prd.gitlab.com
StorageProd 10.70.2.0/23 Storage stor.prd.gitlab.com

Canary

Virtual Network Name: GitLabCanary

Resource Group: GitLabCanary

IP space: 10.192.0.0/13

Subnet Name Subnet Range Tier Domain
ExternalLBCanary 10.192.1.0/24 Load balancers lb.cny.gitlab.com
InternalLBCanary 10.192.2.0/24 Load balancers lb.cny.gitlab.com
APICanary 10.196.2.0/23 Services sv.cny.gitlab.com
GitCanary 10.196.4.0/23 Services sv.cny.gitlab.com
SidekiqCanary 10.196.6.0/23 Services sv.cny.gitlab.com
WebCanary 10.196.8.0/23 Services sv.cny.gitlab.com
RegistryCanary 10.196.10.0/23 Services sv.cny.gitlab.com

Staging

Virtual Network Name: GitLabStaging

Resource Group: GitLabStaging

IP space: 10.128.0.0/12

Resource Group Subnet Range Tier Domain
ExternalLBStaging 10.128.1.0/24 Load balancers lb.stg.gitlab.com
InternalLBStaging 10.128.2.0/24 Load balancers lb.stg.gitlab.com
DBStaging 10.129.1.0/24 Databases db.stg.gitlab.com
RedisStaging 10.129.2.0/24 Databases db.stg.gitlab.com
ElasticSearchStaging 10.129.3.0/24 Databases db.stg.gitlab.com
ConsulStaging 10.130.1.0/24 Support Services infra.stg.gitlab.net
VaultStaging 10.130.2.0/24 Support Services infra.stg.gitlab.net
DeployStaging 10.130.3.0/24 Support Services infra.stg.gitlab.net
LogStaging 10.131.2.0/24 Logging log.stg.gitlab.net
APIStaging 10.132.2.0/23 Services sv.stg.gitlab.com
GitStaging 10.132.4.0/23 Services sv.stg.gitlab.com
SidekiqStaging 10.132.6.0/23 Services sv.stg.gitlab.com
WebStaging 10.132.8.0/23 Services sv.stg.gitlab.com
RegistryStaging 10.132.10.0/23 Services sv.stg.gitlab.com
StorageStaging 10.133.2.0/23 Storage stor.stg.gitlab.com

Azure

The main portion of GitLab.com is hosted on Microsoft Azure. We have the following servers there.

Note that these numbers can fluctuate to adapt to the platform needs.

We also use availability sets to ensure that a minimum number of servers in each group are available at any given time. This ensures that Azure will not reboot all instances in the same availability set at the same time for anything that is planned.

All our servers run the latest Ubuntu LTS unless there is a specific need to do otherwise. Every server is configured with a fully fledged set of firewall rules for increased security.

Load Balancers

We utilize Azure load balancers in front of our HAProxy nodes. This allows us to leverage on the Azure infrastructure for HA as well as taking advantage of the power of HAProxy.

Additionally, we utilize an Azure load balancer to manage PostgreSQL failovers.

Service nodes

Different services have different resource utilization patterns so we use a variety of instance types across our service nodes that are consistent for each group. We have recently isolated traffic by type on dedicated pools of nodes. We hope you noticed the performance improvement.

Digital Ocean

Digital Ocean houses several servers that do not need to directly interact with our main infrastructure. There are many of these that do a variety of things, however not all will be listed here.

The primary things on Digital Ocean at this time are:

AWS

We host our DNS with route53 and we have several EC2 instances for various purposes. The servers you will interact with most are listed Below

Google Cloud

We are currently investigating Google Cloud.

Monitoring

See how it's doing, for more information on that, visit the monitoring handbook.

Technology at GitLab

We use a lot of cool (but boring) technologies here at GitLab. Below is a non-exhaustive list of tech we use here.