diff options
| author | Achilleas Pipinellis <axil@gitlab.com> | 2019-08-20 20:22:43 +0200 |
|---|---|---|
| committer | Achilleas Pipinellis <axil@gitlab.com> | 2019-08-20 20:22:43 +0200 |
| commit | e61308ce1d82e12e5087371469baea4a452875d1 (patch) | |
| tree | 62551a3ae4eab75e5af7e3b35358c07a51b2132f /doc/administration | |
| parent | 4f323bb62fbe71a4352de25cab141f361a3fe1a6 (diff) | |
| parent | 2989ed078c1d45b0959dcecb1bc3c8f4740a3c0d (diff) | |
| download | gitlab-ce-docs-patch-71.tar.gz | |
Merge branch 'master' into docs-patch-71docs-patch-71
Diffstat (limited to 'doc/administration')
125 files changed, 6398 insertions, 4582 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index d7a2e13b53e..02de2caf558 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -2,7 +2,7 @@ last_updated: 2019-02-04 --- -# Audit Events **[STARTER]** +# Audit Events **(STARTER)** GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan][ee]. @@ -32,7 +32,7 @@ There are two kinds of events logged: - Instance events scoped to the whole GitLab instance, used by your Compliance team to perform formal audits. -### Group events **[STARTER]** +### Group events **(STARTER)** NOTE: **Note:** You need Owner [permissions] to view the group Audit Events page. @@ -59,7 +59,7 @@ From there, you can see the following actions: - 2FA enforcement/grace period changed - Roles allowed to create project changed -### Project events **[STARTER]** +### Project events **(STARTER)** NOTE: **Note:** You need Maintainer [permissions] or higher to view the project Audit Events page. @@ -73,8 +73,10 @@ From there, you can see the following actions: - User was added to project and with which [permissions] - Permission changes of a user assigned to a project - User was removed from project +- Project export was downloaded +- Project repository was downloaded -### Instance events **[PREMIUM ONLY]** +### Instance events **(PREMIUM ONLY)** > [Introduced][ee-2336] in [GitLab Premium][ee] 9.3. @@ -94,12 +96,13 @@ recorded: - Changed password - Ask for password reset - Grant OAuth access +- Started/stopped user impersonation It is possible to filter particular actions by choosing an audit data type from the filter drop-down. You can further filter by specific group, project or user (for authentication events). - + ### Missing events diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index ef8c8197d6d..18c415b5ff7 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -1,4 +1,4 @@ -# Auditor users **[PREMIUM ONLY]** +# Auditor users **(PREMIUM ONLY)** >[Introduced][ee-998] in [GitLab Premium][eep] 8.17. @@ -52,7 +52,7 @@ section. **Admin Area > Users**. You will find the option of the access level under the 'Access' section. -  +  1. Click **Save changes** or **Create user** for the changes to take effect. diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index e215a0df6ec..2fc9db0632e 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -1,19 +1,34 @@ --- comments: false +type: index --- -# Authentication and Authorization +# GitLab authentication and authorization GitLab integrates with the following external authentication and authorization -providers. +providers: -- [LDAP](ldap.md) Includes Active Directory, Apple Open Directory, Open LDAP, - and 389 Server - - [LDAP for GitLab EE](ldap-ee.md): LDAP additions to GitLab Enterprise Editions **[STARTER ONLY]** -- [OmniAuth](../../integration/omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, - Bitbucket, Facebook, Shibboleth, Crowd, Azure, Authentiq ID, and JWT -- [CAS](../../integration/cas.md) Configure GitLab to sign in using CAS -- [SAML](../../integration/saml.md) Configure GitLab as a SAML 2.0 Service Provider -- [Okta](okta.md) Configure GitLab to sign in using Okta -- [Authentiq](authentiq.md): Enable the Authentiq OmniAuth provider for passwordless authentication -- [Smartcard](smartcard.md) Smartcard authentication **[PREMIUM ONLY]** +- [Auth0](../../integration/auth0.md) +- [Authentiq](authentiq.md) +- [Azure](../../integration/azure.md) +- [Bitbucket Cloud](../../integration/bitbucket.md) +- [CAS](../../integration/cas.md) +- [Crowd](../../integration/crowd.md) +- [Facebook](../../integration/facebook.md) +- [GitHub](../../integration/github.md) +- [GitLab.com](../../integration/gitlab.md) +- [Google](../../integration/google.md) +- [JWT](jwt.md) +- [Kerberos](../../integration/kerberos.md) +- [LDAP](ldap.md): Includes Active Directory, Apple Open Directory, Open LDAP, + and 389 Server. + - [LDAP for GitLab EE](ldap-ee.md): LDAP additions to GitLab Enterprise Editions **(STARTER ONLY)** + - [Google Secure LDAP](google_secure_ldap.md) +- [Okta](okta.md) +- [Salesforce](../../integration/salesforce.md) +- [SAML](../../integration/saml.md) +- [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(SILVER ONLY)** +- [Shibboleth](../../integration/shibboleth.md) +- [Smartcard](smartcard.md) **(PREMIUM ONLY)** +- [Twitter](../../integration/twitter.md) +- [UltraAuth](../../integration/ultra_auth.md) diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index 726622d8599..b84eca4ef0d 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Authentiq OmniAuth Provider To enable the Authentiq OmniAuth provider for passwordless authentication you must register an application with Authentiq. @@ -8,47 +12,48 @@ Authentiq will generate a Client ID and the accompanying Client Secret for you t 1. On your GitLab server, open the configuration file: - For omnibus installation - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + For omnibus installation + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - sudo -u git -H editor /home/git/gitlab/config/gitlab.yml - ``` + ```sh + sudo -u git -H editor /home/git/gitlab/config/gitlab.yml + ``` 1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings to enable single sign-on and add Authentiq as an OAuth provider. 1. Add the provider configuration for Authentiq: - For Omnibus packages: - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "authentiq", - "app_id" => "YOUR_CLIENT_ID", - "app_secret" => "YOUR_CLIENT_SECRET", - "args" => { - "scope": 'aq:name email~rs address aq:push' - } - } - ] - ``` - - For installations from source: - - ```yaml - - { name: 'authentiq', - app_id: 'YOUR_CLIENT_ID', - app_secret: 'YOUR_CLIENT_SECRET', - args: { - scope: 'aq:name email~rs address aq:push' - } - } - ``` + For Omnibus packages: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "authentiq", + "app_id" => "YOUR_CLIENT_ID", + "app_secret" => "YOUR_CLIENT_SECRET", + "args" => { + "scope": 'aq:name email~rs address aq:push' + } + } + ] + ``` + + For installations from source: + + ```yaml + - { name: 'authentiq', + app_id: 'YOUR_CLIENT_ID', + app_secret: 'YOUR_CLIENT_SECRET', + args: { + scope: 'aq:name email~rs address aq:push' + } + } + ``` 1. The `scope` is set to request the user's name, email (required and signed), and permission to send push notifications to sign in on subsequent visits. See [OmniAuth Authentiq strategy](https://github.com/AuthentiqID/omniauth-authentiq/wiki/Scopes,-callback-url-configuration-and-responses) for more information on scopes and modifiers. @@ -65,3 +70,15 @@ On the sign in page there should now be an Authentiq icon below the regular sign - If not they will be prompted to download the app and then follow the procedure above. If everything goes right, the user will be returned to GitLab and will be signed in. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 6db74958d5a..ac63b4f2b97 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -1,60 +1,67 @@ +--- +type: reference +--- + # Atlassian Crowd OmniAuth Provider +Authenticate to GitLab using the Atlassian Crowd OmniAuth provider. + ## Configure a new Crowd application 1. Choose 'Applications' in the top menu, then 'Add application'. 1. Go through the 'Add application' steps, entering the appropriate details. The screenshot below shows an example configuration. -  +  ## Configure GitLab 1. On your GitLab server, open the configuration file. - **Omnibus:** + **Omnibus:** - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - **Source:** + **Source:** - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` 1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. 1. Add the provider configuration: - **Omnibus:** - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "crowd", - "args" => { - "crowd_server_url" => "CROWD_SERVER_URL", - "application_name" => "YOUR_APP_NAME", - "application_password" => "YOUR_APP_PASSWORD" - } - } - ] - ``` - - **Source:** - - ``` - - { name: 'crowd', - args: { - crowd_server_url: 'CROWD_SERVER_URL', - application_name: 'YOUR_APP_NAME', - application_password: 'YOUR_APP_PASSWORD' } } - ``` + **Omnibus:** + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "crowd", + "args" => { + "crowd_server_url" => "CROWD_SERVER_URL", + "application_name" => "YOUR_APP_NAME", + "application_password" => "YOUR_APP_PASSWORD" + } + } + ] + ``` + + **Source:** + + ``` + - { name: 'crowd', + args: { + crowd_server_url: 'CROWD_SERVER_URL', + application_name: 'YOUR_APP_NAME', + application_password: 'YOUR_APP_PASSWORD' } } + ``` + 1. Change `CROWD_SERVER_URL` to the URL of your Crowd server. 1. Change `YOUR_APP_NAME` to the application name from Crowd applications page. 1. Change `YOUR_APP_PASSWORD` to the application password you've set. @@ -77,4 +84,4 @@ could not authorize you from Crowd because invalid credentials Please make sure the Crowd users who need to login to GitLab are authorized to [the application](#configure-a-new-crowd-application) in the step of **Authorisation**. This could be verified by try "Authentication test" for Crowd as of 2.11. -
\ No newline at end of file + diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md index 760af0cfd1a..55e6f53622c 100644 --- a/doc/administration/auth/google_secure_ldap.md +++ b/doc/administration/auth/google_secure_ldap.md @@ -1,4 +1,8 @@ -# Google Secure LDAP **[CORE ONLY]** +--- +type: reference +--- + +# Google Secure LDAP **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/46391) in GitLab 11.9. @@ -13,7 +17,7 @@ The steps below cover: ## Configuring Google LDAP client -1. Navigate to https://admin.google.com and sign in as a GSuite domain administrator. +1. Navigate to <https://admin.google.com/Dashboard> and sign in as a GSuite domain administrator. 1. Go to **Apps > LDAP > Add Client**. @@ -66,7 +70,7 @@ values obtained during the LDAP client configuration earlier: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby + ```ruby gitlab_rails['ldap_enabled'] = true gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below main: # 'main' is the GitLab 'provider ID' of this LDAP server @@ -127,7 +131,7 @@ values obtained during the LDAP client configuration earlier: AcZSFJQjdg5BTyvdEDhaYUKGdRw= -----END PRIVATE KEY----- EOS - ``` + ``` 1. Save the file and [reconfigure] GitLab for the changes to take effect. @@ -137,7 +141,7 @@ values obtained during the LDAP client configuration earlier: 1. Edit `config/gitlab.yml`: - ```yaml + ```yaml ldap: enabled: true servers: @@ -202,6 +206,17 @@ values obtained during the LDAP client configuration earlier: 1. Save the file and [restart] GitLab for the changes to take effect. - [reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [restart]: ../restart_gitlab.md#installations-from-source + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 1f67e8f5744..86dd398343b 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -1,15 +1,9 @@ --- -author: Chris Wilson -author_gitlab: MrChrisW -level: intermediary -article_type: admin guide -date: 2017-05-03 +type: howto --- # How to configure LDAP with GitLab CE -## Introduction - Managing a large number of users in GitLab can become a burden for system administrators. As an organization grows so do user accounts. Keeping these user accounts in sync across multiple enterprise applications often becomes a time consuming task. In this guide we will focus on configuring GitLab with Active Directory. [Active Directory](https://en.wikipedia.org/wiki/Active_Directory) is a popular LDAP compatible directory service provided by Microsoft, included in all modern Windows Server operating systems. @@ -111,7 +105,7 @@ The initial configuration of LDAP in GitLab requires changes to the `gitlab.rb` The two Active Directory specific values are `active_directory: true` and `uid: 'sAMAccountName'`. `sAMAccountName` is an attribute returned by Active Directory used for GitLab usernames. See the example output from `ldapsearch` for a full list of attributes a "person" object (user) has in **AD** - [`ldapsearch` example](#using-ldapsearch-unix) -> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](../how_to_configure_ldap_gitlab_ee/index.html#gitlab-enterprise-edition---ldap-features) **[STARTER ONLY]** +> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](../how_to_configure_ldap_gitlab_ee/index.html#gitlab-enterprise-edition---ldap-features) **(STARTER ONLY)** ### Example `gitlab.rb` LDAP @@ -267,4 +261,16 @@ have extended functionalities with LDAP, such as: - Updating user permissions - Multiple LDAP servers -Read through the article on [LDAP for GitLab EE](../how_to_configure_ldap_gitlab_ee/index.md) **[STARTER ONLY]** for an overview. +Read through the article on [LDAP for GitLab EE](../how_to_configure_ldap_gitlab_ee/index.md) **(STARTER ONLY)** for an overview. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md index 4d82a7370bb..366acb9ed3e 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md @@ -1,16 +1,10 @@ --- -author: Chris Wilson -author_gitlab: MrChrisW -level: intermediary -article_type: admin guide -date: 2017-05-03 +type: howto --- -# How to configure LDAP with GitLab EE **[STARTER ONLY]** +# How to configure LDAP with GitLab EE **(STARTER ONLY)** -## Introduction - -The present article follows [How to Configure LDAP with GitLab CE](../how_to_configure_ldap_gitlab_ce/index.md). Make sure to read through it before moving forward. +This article expands on [How to Configure LDAP with GitLab CE](../how_to_configure_ldap_gitlab_ce/index.md). Make sure to read through it before moving forward. ## GitLab Enterprise Edition - LDAP features @@ -117,3 +111,15 @@ Integration of GitLab with Active Directory (LDAP) reduces the complexity of use It has the advantage of improving user permission controls, whilst easing the deployment of GitLab into an existing [IT environment](https://www.techopedia.com/definition/29199/it-infrastructure). GitLab EE offers advanced group management and multiple LDAP servers. With the assistance of the [GitLab Support](https://about.gitlab.com/support) team, setting up GitLab with an existing AD/LDAP solution will be a smooth and painless process. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_1.png b/doc/administration/auth/img/google_secure_ldap_add_step_1.png Binary files differindex fd254443d75..bee9c602a14 100644 --- a/doc/administration/auth/img/google_secure_ldap_add_step_1.png +++ b/doc/administration/auth/img/google_secure_ldap_add_step_1.png diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_2.png b/doc/administration/auth/img/google_secure_ldap_add_step_2.png Binary files differindex 611a21ae03c..b127410cb8c 100644 --- a/doc/administration/auth/img/google_secure_ldap_add_step_2.png +++ b/doc/administration/auth/img/google_secure_ldap_add_step_2.png diff --git a/doc/administration/auth/img/google_secure_ldap_client_settings.png b/doc/administration/auth/img/google_secure_ldap_client_settings.png Binary files differindex 3c0b3f3d4bd..868e6645f56 100644 --- a/doc/administration/auth/img/google_secure_ldap_client_settings.png +++ b/doc/administration/auth/img/google_secure_ldap_client_settings.png diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 497298503ad..e6b3287ce60 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -1,67 +1,71 @@ +--- +type: reference +--- + # JWT OmniAuth provider To enable the JWT OmniAuth provider, you must register your application with JWT. JWT will provide you with a secret key for you to use. -1. On your GitLab server, open the configuration file. - - For Omnibus GitLab: - - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` - - For installations from source: - - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` - -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. -1. Add the provider configuration. - - For Omnibus GitLab: - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { name: 'jwt', - args: { - secret: 'YOUR_APP_SECRET', - algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' - uid_claim: 'email', - required_claims: ['name', 'email'], - info_maps: { name: 'name', email: 'email' }, - auth_url: 'https://example.com/', - valid_within: 3600 # 1 hour - } - } - ] - ``` - - For installation from source: - - ``` - - { name: 'jwt', - args: { - secret: 'YOUR_APP_SECRET', - algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' - uid_claim: 'email', - required_claims: ['name', 'email'], - info_map: { name: 'name', email: 'email' }, - auth_url: 'https://example.com/', - valid_within: 3600 # 1 hour - } - } - ``` - - NOTE: **Note:** For more information on each configuration option refer to - the [OmniAuth JWT usage documentation](https://github.com/mbleigh/omniauth-jwt#usage). - -1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL. -1. Save the configuration file. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. On your GitLab server, open the configuration file. + + For Omnibus GitLab: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. +1. Add the provider configuration. + + For Omnibus GitLab: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { name: 'jwt', + args: { + secret: 'YOUR_APP_SECRET', + algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' + uid_claim: 'email', + required_claims: ['name', 'email'], + info_maps: { name: 'name', email: 'email' }, + auth_url: 'https://example.com/', + valid_within: 3600 # 1 hour + } + } + ] + ``` + + For installation from source: + + ``` + - { name: 'jwt', + args: { + secret: 'YOUR_APP_SECRET', + algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' + uid_claim: 'email', + required_claims: ['name', 'email'], + info_map: { name: 'name', email: 'email' }, + auth_url: 'https://example.com/', + valid_within: 3600 # 1 hour + } + } + ``` + + NOTE: **Note:** For more information on each configuration option refer to + the [OmniAuth JWT usage documentation](https://github.com/mbleigh/omniauth-jwt#usage). + +1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL. +1. Save the configuration file. +1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a JWT icon below the regular sign in form. Click the icon to begin the authentication process. JWT will ask the user to @@ -70,3 +74,15 @@ will be redirected to GitLab and will be signed in. [reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [restart GitLab]: ../restart_gitlab.md#installations-from-source + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 30095d35705..34f3cfa353f 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -1,36 +1,30 @@ -# LDAP Additions in GitLab EE **[STARTER ONLY]** - -This is a continuation of the main [LDAP documentation](ldap.md), detailing LDAP -features specific to GitLab Enterprise Edition Starter, Premium and Ultimate. +--- +type: reference +--- -## Overview +# LDAP Additions in GitLab EE **(STARTER ONLY)** -[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) -stands for **Lightweight Directory Access Protocol**, which -is a standard application protocol for -accessing and maintaining distributed directory information services -over an Internet Protocol (IP) network. +This section documents LDAP features specific to to GitLab Enterprise Edition +[Starter](https://about.gitlab.com/pricing/#self-managed) and above. -GitLab integrates with LDAP to support **user authentication**. This integration -works with most LDAP-compliant directory servers, including Microsoft Active -Directory, Apple Open Directory, Open LDAP, and 389 Server. -**GitLab Enterprise Edition** includes enhanced integration, including group -membership syncing. +For documentation relevant to both Community Edition and Enterprise Edition, +see the main [LDAP documentation](ldap.md). ## Use cases -- User sync: Once a day, GitLab will update users against LDAP +- User sync: Once a day, GitLab will update users against LDAP. - Group sync: Once an hour, GitLab will update group membership - based on LDAP group members + based on LDAP group members. -## Multiple LDAP servers +## Multiple LDAP servers **(STARTER ONLY)** With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers that your GitLab instance will connect to. -To add another LDAP server, you can start by duplicating the settings under -[the main configuration](ldap.md#configuration) and edit them to match the -additional LDAP server. +To add another LDAP server: + +1. Duplicating the settings under [the main configuration](ldap.md#configuration). +1. Edit them to match the additional LDAP server. Be sure to choose a different provider ID made of letters a-z and numbers 0-9. This ID will be stored in the database so that GitLab can remember which LDAP @@ -43,19 +37,19 @@ users against LDAP. The process will execute the following access checks: -1. Ensure the user is still present in LDAP -1. If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This will only be checked if - `active_directory: true` is set in the LDAP configuration [^1] +- Ensure the user is still present in LDAP. +- If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This will only be checked if + `active_directory: true` is set in the LDAP configuration. [^1] The user will be set to `ldap_blocked` state in GitLab if the above conditions fail. This means the user will not be able to login or push/pull code. The process will also update the following user information: -1. Email address -1. If `sync_ssh_keys` is set, SSH public keys -1. If Kerberos is enabled, Kerberos identity +- Email address. +- If `sync_ssh_keys` is set, SSH public keys. +- If Kerberos is enabled, Kerberos identity. NOTE: **Note:** The LDAP sync process updates existing users while new users will @@ -68,9 +62,6 @@ first time GitLab will trigger a sync for groups the user should be a member of. That way they don't need to wait for the hourly sync to be granted access to their groups and projects. -In GitLab Premium, we can also add a GitLab group to sync with one or multiple LDAP groups or we can -also add a filter. The filter must comply with the syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). - A group sync process will run every hour on the hour, and `group_base` must be set in LDAP configuration for LDAP synchronizations based on group CN to work. This allows GitLab group membership to be automatically updated based on LDAP group members. @@ -85,19 +76,19 @@ following. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## Base where we can search for groups - ## - ## Ex. ou=groups,dc=gitlab,dc=example - ## - ## - group_base: ou=groups,dc=example,dc=com - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + EOS + ``` 1. [Reconfigure GitLab][reconfigure] for the changes to take effect. @@ -105,24 +96,37 @@ following. 1. Edit `/home/git/gitlab/config/gitlab.yml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - group_base: ou=groups,dc=example,dc=com - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + ``` 1. [Restart GitLab][restart] for the changes to take effect. ---- - To take advantage of group sync, group owners or maintainers will need to create an -LDAP group link in their group **Settings > LDAP Groups** page. Multiple LDAP -groups and/or filters can be linked with a single GitLab group. When the link is -created, an access level/role is specified (Guest, Reporter, Developer, Maintainer, -or Owner). +LDAP group link in their group **Settings > LDAP Groups** page. + +Multiple LDAP groups and [filters](#filters-premium-only) can be linked with +a single GitLab group. When the link is created, an access level/role is +specified (Guest, Reporter, Developer, Maintainer, or Owner). + +### Filters **(PREMIUM ONLY)** + +In GitLab Premium, you can add an LDAP user filter for group synchronization. +Filters allow for complex logic without creating a special LDAP group. + +To sync GitLab group membership based on an LDAP filter: + +1. Open the **LDAP Synchronization** page for the GitLab group. +1. Select **LDAP user filter** as the **Sync method**. +1. Enter an LDAP user filter in the **LDAP user filter** field. + +The filter must comply with the +syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). ## Administrator sync @@ -140,30 +144,30 @@ group, as opposed to the full DN. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## Base where we can search for groups - ## - ## Ex. ou=groups,dc=gitlab,dc=example - ## - ## - group_base: ou=groups,dc=example,dc=com - - ## - ## The CN of a group containing GitLab administrators - ## - ## Ex. administrators - ## - ## Note: Not `cn=administrators` or the full DN - ## - ## - admin_group: my_admin_group - - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + + ## + ## The CN of a group containing GitLab administrators + ## + ## Ex. administrators + ## + ## Note: Not `cn=administrators` or the full DN + ## + ## + admin_group: my_admin_group + + EOS + ``` 1. [Reconfigure GitLab][reconfigure] for the changes to take effect. @@ -171,40 +175,51 @@ group, as opposed to the full DN. 1. Edit `/home/git/gitlab/config/gitlab.yml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - group_base: ou=groups,dc=example,dc=com - admin_group: my_admin_group - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + admin_group: my_admin_group + ``` 1. [Restart GitLab][restart] for the changes to take effect. +## Global group memberships lock + +"Lock memberships to LDAP synchronization" setting allows instance administrators +to lock down user abilities to invite new members to a group. + +When enabled, the following applies: + +- Only administrator can manage memberships of any group including access levels. +- Users are not allowed to share project with other groups or invite members to + a project created in a group. + ## Adjusting LDAP user sync schedule > Introduced in GitLab Enterprise Edition Starter. NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create -these values, for example http://www.crontabgenerator.com/. +these values, for example <http://www.crontabgenerator.com/>. By default, GitLab will run a worker once per day at 01:30 a.m. server time to check and update GitLab users against LDAP. You can manually configure LDAP user sync times by setting the following configuration values. The example below shows how to set LDAP user -sync to run once every 12 hours at the top of the hour. +sync to run once every 12 hours at the top of the hour. **Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" - ``` + ```ruby + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -212,11 +227,11 @@ sync to run once every 12 hours at the top of the hour. 1. Edit `config/gitlab.yaml`: - ```yaml - cron_jobs: - ldap_sync_worker_cron: - "0 */12 * * *" - ``` + ```yaml + cron_jobs: + ldap_sync_worker_cron: + "0 */12 * * *" + ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -224,7 +239,7 @@ sync to run once every 12 hours at the top of the hour. NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create -these values, for example http://www.crontabgenerator.com/. +these values, for example <http://www.crontabgenerator.com/>. By default, GitLab will run a group sync process every hour, on the hour. @@ -236,16 +251,16 @@ for installations with a large number of LDAP users. Please review the your installation compares before proceeding. You can manually configure LDAP group sync times by setting the -following configuration values. The example below shows how to set group -sync to run once every 2 hours at the top of the hour. +following configuration values. The example below shows how to set group +sync to run once every 2 hours at the top of the hour. **Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" - ``` + ```ruby + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -253,11 +268,11 @@ sync to run once every 2 hours at the top of the hour. 1. Edit `config/gitlab.yaml`: - ```yaml - cron_jobs: - ldap_group_sync_worker_cron: - "*/30 * * * *" - ``` + ```yaml + cron_jobs: + ldap_group_sync_worker_cron: + "*/30 * * * *" + ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -274,20 +289,20 @@ task. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## An array of CNs of groups containing users that should be considered external - ## - ## Ex. ['interns', 'contractors'] - ## - ## Note: Not `cn=interns` or the full DN - ## - external_groups: ['interns', 'contractors'] - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## An array of CNs of groups containing users that should be considered external + ## + ## Ex. ['interns', 'contractors'] + ## + ## Note: Not `cn=interns` or the full DN + ## + external_groups: ['interns', 'contractors'] + EOS + ``` 1. [Reconfigure GitLab][reconfigure] for the changes to take effect. @@ -295,14 +310,14 @@ task. 1. Edit `config/gitlab.yaml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - external_groups: ['interns', 'contractors'] - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + external_groups: ['interns', 'contractors'] + ``` 1. [Restart GitLab][restart] for the changes to take effect. @@ -321,10 +336,18 @@ administrative duties. ### Supported LDAP group types/attributes -GitLab supports LDAP groups that use member attributes `member`, `submember`, -`uniquemember`, `memberof` and `memberuid`. This means group sync supports, at -least, LDAP groups with object class `groupOfNames`, `posixGroup`, and -`groupOfUniqueName`. Other object classes should work fine as long as members +GitLab supports LDAP groups that use member attributes: + +- `member` +- `submember` +- `uniquemember` +- `memberof` +- `memberuid`. + +This means group sync supports, at least, LDAP groups with object class: +`groupOfNames`, `posixGroup`, and `groupOfUniqueName`. + +Other object classes should work fine as long as members are defined as one of the mentioned attributes. This also means GitLab supports Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. Other LDAP servers should work, too. @@ -332,11 +355,12 @@ Other LDAP servers should work, too. Active Directory also supports nested groups. Group sync will recursively resolve membership if `active_directory: true` is set in the configuration file. -> **Note:** Nested group membership will only be resolved if the nested group - also falls within the configured `group_base`. For example, if GitLab sees a - nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but - the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` - will be ignored. +NOTE: **Note:** +Nested group membership will only be resolved if the nested group +also falls within the configured `group_base`. For example, if GitLab sees a +nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but +the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` +will be ignored. ### Queries @@ -391,7 +415,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server [^1]: In Active Directory, a user is marked as disabled/blocked if the user account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) - has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/ + has bit 2 set. See <https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/> for more information. ### User DN has changed @@ -411,10 +435,10 @@ things to check to debug the situation. - Ensure LDAP configuration has a `group_base` specified. This configuration is required for group sync to work properly. - Ensure the correct LDAP group link is added to the GitLab group. Check group - links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**. -- Check that the user has an LDAP identity + links by visiting the GitLab group, then **Settings dropdown > LDAP groups**. +- Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. Navigate to **Admin area -> Users**. + 1. Navigate to **Admin area > Users**. 1. Search for the user 1. Open the user, by clicking on their name. Do not click 'Edit'. 1. Navigate to the **Identities** tab. There should be an LDAP identity with @@ -425,68 +449,74 @@ Often, the best way to learn more about why group sync is behaving a certain way is to enable debug logging. There is verbose output that details every step of the sync. -1. Start a Rails console +1. Start a Rails console: - ```bash - # For Omnibus installations - sudo gitlab-rails console + ```bash + # For Omnibus installations + sudo gitlab-rails console + + # For installations from source + sudo -u git -H bundle exec rails console production + ``` - # For installations from source - sudo -u git -H bundle exec rails console production - ``` 1. Set the log level to debug (only for this session): - ```ruby - Rails.logger.level = Logger::DEBUG - ``` + ```ruby + Rails.logger.level = Logger::DEBUG + ``` + 1. Choose a GitLab group to test with. This group should have an LDAP group link already configured. If the output is `nil`, the group could not be found. If a bunch of group attributes are output, your group was found successfully. - ```ruby - group = Group.find_by(name: 'my_group') + ```ruby + group = Group.find_by(name: 'my_group') + + # Output + => #<Group:0x007fe825196558 id: 1234, name: "my_group"...> + ``` - # Output - => #<Group:0x007fe825196558 id: 1234, name: "my_group"...> - ``` 1. Run a group sync for this particular group. - ```ruby - EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group) - ``` + ```ruby + EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group) + ``` + 1. Look through the output of the sync. See [example log output](#example-log-output) below for more information about the output. 1. If you still aren't able to see why the user isn't being added, query the LDAP group directly to see what members are listed. Still in the Rails console, run the following query: - ```ruby - adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider - ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter) + ```ruby + adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider + ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter) + + # Output + => #<EE::Gitlab::Auth::LDAP::Group:0x007fcbdd0bb6d8 + ``` - # Output - => #<EE::Gitlab::Auth::LDAP::Group:0x007fcbdd0bb6d8 - ``` 1. Query the LDAP group's member DNs and see if the user's DN is in the list. One of the DNs here should match the 'Identifier' from the LDAP identity checked earlier. If it doesn't, the user does not appear to be in the LDAP group. - ```ruby - ldap_group.member_dns + ```ruby + ldap_group.member_dns + + # Output + => ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"] + ``` - # Output - => ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"] - ``` 1. Some LDAP servers don't store members by DN. Rather, they use UIDs instead. If you didn't see results from the last query, try querying by UIDs instead. - ```ruby - ldap_group.member_uids + ```ruby + ldap_group.member_uids - # Output - => ['john','mary'] - ``` + # Output + => ['john','mary'] + ``` #### Example log output @@ -523,8 +553,9 @@ and more DNs may be added, or existing entries modified, based on additional LDAP group lookups. The very last occurrence of this entry should indicate exactly which users GitLab believes should be added to the group. -> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' - and 50 is 'Owner' +NOTE: **Note:** +10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' +and 50 is 'Owner'. ```bash Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 54279897e04..186bf4c4825 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -1,32 +1,50 @@ -[//]: # (Do *NOT* modify this file in EE documentation. All changes in this) -[//]: # (file should happen in CE, too. If the change is EE-specific, put) -[//]: # (it in `ldap-ee.md`.) +--- +type: reference +--- + +<!-- If the change is EE-specific, put it in `ldap-ee.md`, NOT here. --> # LDAP GitLab integrates with LDAP to support user authentication. -This integration works with most LDAP-compliant directory -servers, including Microsoft Active Directory, Apple Open Directory, Open LDAP, -and 389 Server. GitLab Enterprise Editions include enhanced integration, + +This integration works with most LDAP-compliant directory servers, including: + +- Microsoft Active Directory +- Apple Open Directory +- Open LDAP +- 389 Server. + +GitLab Enterprise Editions (EE) include enhanced integration, including group membership syncing as well as multiple LDAP servers support. -## GitLab EE +For more details about EE-specific LDAP features, see the +[LDAP Enterprise Edition documentation](ldap-ee.md). -The information on this page is relevant for both GitLab CE and EE. For more -details about EE-specific LDAP features, see the -[LDAP Enterprise Edition documentation](ldap-ee.md). **[STARTER ONLY]** +NOTE: **Note:** +The information on this page is relevant for both GitLab CE and EE. + +## Overview + +[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +stands for **Lightweight Directory Access Protocol**, which is a standard +application protocol for accessing and maintaining distributed directory +information services over an Internet Protocol (IP) network. ## Security -GitLab assumes that LDAP users are not able to change their LDAP 'mail', 'email' -or 'userPrincipalName' attribute. An LDAP user who is allowed to change their -email on the LDAP server can potentially -[take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) -on your GitLab server. +GitLab assumes that LDAP users: + +- Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attribute. + An LDAP user who is allowed to change their email on the LDAP server can potentially + [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) + on your GitLab server. +- Have unique email addresses, otherwise it is possible for LDAP users with the same + email address to share the same GitLab account. We recommend against using LDAP integration if your LDAP users are -allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on -the LDAP server. +allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on +the LDAP server or share email addresses. ### User deletion @@ -48,7 +66,7 @@ LDAP-enabled users can always authenticate with Git using their GitLab username or email and LDAP password, even if password authentication for Git is disabled in the application settings. -## Google Secure LDAP **[CORE ONLY]** +## Google Secure LDAP **(CORE ONLY)** > Introduced in GitLab 11.9. @@ -62,9 +80,12 @@ NOTE: **Note**: In GitLab Enterprise Edition Starter, you can configure multiple LDAP servers to connect to one GitLab server. -For a complete guide on configuring LDAP with GitLab Community Edition, please check -the admin guide [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). -For GitLab Enterprise Editions, see also [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **[STARTER ONLY]** +For a complete guide on configuring LDAP with: + +- GitLab Community Edition, see + [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). +- Enterprise Editions, see + [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)** To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus @@ -382,35 +403,48 @@ production: Tip: If you want to limit access to the nested members of an Active Directory group, you can use the following syntax: -``` +```text (memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) ``` Find more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter at <https://docs.microsoft.com/en-us/windows/desktop/ADSI/search-filter-syntax>. Support for nested members in the user filter should not be confused with -[group sync nested groups support](ldap-ee.md#supported-ldap-group-typesattributes). **[STARTER ONLY]** +[group sync nested groups support](ldap-ee.md#supported-ldap-group-typesattributes). **(STARTER ONLY)** Please note that GitLab does not support the custom filter syntax used by omniauth-ldap. ### Escaping special characters -If the `user_filter` DN contains special characters. For example, a comma: +The `user_filter` DN can contain special characters. For example: -``` -OU=GitLab, Inc,DC=gitlab,DC=com -``` +- A comma: -This character needs to be escaped as documented in [RFC 4515](https://tools.ietf.org/search/rfc4515). + ```text + OU=GitLab, Inc,DC=gitlab,DC=com + ``` -Due to the way the string is parsed, the special character needs to be converted -to hex and `\\5C\\` (`5C` = `\` in hex) added before it. -As an example the above DN would look like +- Open and close brackets: -``` -OU=GitLab\\5C\\2C Inc,DC=gitlab,DC=com -``` + ```text + OU=Gitlab (Inc),DC=gitlab,DC=com + ``` + + These characters must be escaped as documented in + [RFC 4515](https://tools.ietf.org/search/rfc4515). + +- Escape commas with `\2C`. For example: + + ```text + OU=GitLab\2C Inc,DC=gitlab,DC=com + ``` + +- Escape open and close brackets with `\28` and `\29`, respectively. For example: + + ```text + OU=Gitlab \28Inc\29,DC=gitlab,DC=com + ``` ## Enabling LDAP sign-in for existing GitLab users @@ -434,13 +468,13 @@ the configuration option `lowercase_usernames`. By default, this configuration o 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - # snip... - lowercase_usernames: true - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + # snip... + lowercase_usernames: true + EOS + ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -448,14 +482,14 @@ the configuration option `lowercase_usernames`. By default, this configuration o 1. Edit `config/gitlab.yaml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - lowercase_usernames: true - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + lowercase_usernames: true + ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -483,13 +517,20 @@ be mandatory and clients cannot be authenticated with the TLS protocol. ## Troubleshooting +If a user account is blocked or unblocked due to the LDAP configuration, a +message will be logged to `application.log`. + +If there is an unexpected error during an LDAP lookup (configuration error, +timeout), the login is rejected and a message will be logged to +`production.log`. + ### Debug LDAP user filter with ldapsearch -This example uses ldapsearch and assumes you are using ActiveDirectory. The +This example uses `ldapsearch` and assumes you are using ActiveDirectory. The following query returns the login names of the users that will be allowed to log in to GitLab if you configure your own user_filter. -``` +```sh ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName ``` @@ -508,26 +549,17 @@ ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$ba - Run the following check command to make sure that the LDAP settings are correct and GitLab can see your users: - ```bash - # For Omnibus installations - sudo gitlab-rake gitlab:ldap:check + ```bash + # For Omnibus installations + sudo gitlab-rake gitlab:ldap:check - # For installations from source - sudo -u git -H bundle exec rake gitlab:ldap:check RAILS_ENV=production - ``` + # For installations from source + sudo -u git -H bundle exec rake gitlab:ldap:check RAILS_ENV=production + ``` -### Connection Refused +### Connection refused If you are getting 'Connection Refused' errors when trying to connect to the LDAP server please double-check the LDAP `port` and `encryption` settings used by GitLab. Common combinations are `encryption: 'plain'` and `port: 389`, OR `encryption: 'simple_tls'` and `port: 636`. - -### Troubleshooting - -If a user account is blocked or unblocked due to the LDAP configuration, a -message will be logged to `application.log`. - -If there is an unexpected error during an LDAP lookup (configuration error, -timeout), the login is rejected and a message will be logged to -`production.log`. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 00422ec347c..3445f916ef4 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # OpenID Connect OmniAuth provider GitLab can use [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0.html) as an OmniAuth provider. @@ -5,101 +9,103 @@ GitLab can use [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0 To enable the OpenID Connect OmniAuth provider, you must register your application with an OpenID Connect provider. The OpenID Connect will provide you with a client details and secret for you to use. -1. On your GitLab server, open the configuration file. - - For Omnibus GitLab: - - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` - - For installations from source: - - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` - - See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. - -1. Add the provider configuration. - - For Omnibus GitLab: - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { 'name' => 'openid_connect', - 'label' => '<your_oidc_label>', - 'args' => { - "name' => 'openid_connect', - 'scope' => ['openid','profile'], - 'response_type' => 'code', - 'issuer' => '<your_oidc_url>', - 'discovery' => true, - 'client_auth_method' => 'query', - 'uid_field' => '<uid_field>', - 'client_options' => { - 'identifier' => '<your_oidc_client_id>', - 'secret' => '<your_oidc_client_secret>', - 'redirect_uri' => '<your_gitlab_url>/users/auth/openid_connect/callback' - } - } - } - ] - ``` - - For installation from source: - - ```yaml - - { name: 'openid_connect', - label: '<your_oidc_label>', - args: { - name: 'openid_connect', - scope: ['openid','profile'], - response_type: 'code', - issuer: '<your_oidc_url>', - discovery: true, - client_auth_method: 'query', - uid_field: '<uid_field>', - client_options: { - identifier: '<your_oidc_client_id>', - secret: '<your_oidc_client_secret>', - redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback' - } - } - } - ``` - - > **Note:** - > - > - For more information on each configuration option refer to - the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) and - the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). +1. On your GitLab server, open the configuration file. + + For Omnibus GitLab: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + + See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration. + + For Omnibus GitLab: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { 'name' => 'openid_connect', + 'label' => '<your_oidc_label>', + 'args' => { + "name' => 'openid_connect', + 'scope' => ['openid','profile'], + 'response_type' => 'code', + 'issuer' => '<your_oidc_url>', + 'discovery' => true, + 'client_auth_method' => 'query', + 'uid_field' => '<uid_field>', + 'client_options' => { + 'identifier' => '<your_oidc_client_id>', + 'secret' => '<your_oidc_client_secret>', + 'redirect_uri' => '<your_gitlab_url>/users/auth/openid_connect/callback' + } + } + } + ] + ``` + + For installation from source: + + ```yaml + - { name: 'openid_connect', + label: '<your_oidc_label>', + args: { + name: 'openid_connect', + scope: ['openid','profile'], + response_type: 'code', + issuer: '<your_oidc_url>', + discovery: true, + client_auth_method: 'query', + uid_field: '<uid_field>', + client_options: { + identifier: '<your_oidc_client_id>', + secret: '<your_oidc_client_secret>', + redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback' + } + } + } + ``` + + NOTE: **Note:** + For more information on each configuration option refer to the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) + and the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). 1. For the configuration above, change the values for the provider to match your OpenID Connect client setup. Use the following as a guide: - - `<your_oidc_label>` is the label that will be displayed on the login page. - - `<your_oidc_url>` (optional) is the URL that points to the OpenID Connect provider. For example, `https://example.com/auth/realms/your-realm`. - If this value is not provided, the URL is constructed from the `client_options` in the following format: `<client_options.scheme>://<client_options.host>:<client_options.port>`. - - If `discovery` is set to `true`, the OpenID Connect provider will try to auto discover the client options using `<your_oidc_url>/.well-known/openid-configuration`. Defaults to `false`. - - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. - If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. - - `client_options` are the OpenID Connect client-specific options. Specifically: - - - `identifier` is the client identifier as configured in the OpenID Connect service provider. - - `secret` is the client secret as configured in the OpenID Connect service provider. - - `redirect_uri` is the GitLab URL to redirect the user to after successful login. For example, `http://example.com/users/auth/openid_connect/callback`. - - `end_session_endpoint` (optional) is the URL to the endpoint that end the session (logout). Can be provided if auto-discovery disabled or unsuccessful. - - The following `client_options` are optional unless auto-discovery is disabled or unsuccessful: - - - `authorization_endpoint` is the URL to the endpoint that authorizes the end user. - - `token_endpoint` is the URL to the endpoint that provides Access Token. - - `userinfo_endpoint` is the URL to the endpoint that provides the user information. - - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. - -1. Save the configuration file. -1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) - for the changes to take effect if you installed GitLab via Omnibus or from source respectively. + - `<your_oidc_label>` is the label that will be displayed on the login page. + - `<your_oidc_url>` (optional) is the URL that points to the OpenID Connect provider. For example, `https://example.com/auth/realms/your-realm`. + If this value is not provided, the URL is constructed from the `client_options` in the following format: `<client_options.scheme>://<client_options.host>:<client_options.port>`. + - If `discovery` is set to `true`, the OpenID Connect provider will try to auto discover the client options using `<your_oidc_url>/.well-known/openid-configuration`. Defaults to `false`. + - `client_auth_method` (optional) specifies the method used for authenticating the client with the OpenID Connect provider. + - Supported values are: + - `basic` - HTTP Basic Authentication + - `jwt_bearer` - JWT based authentication (private key and client secret signing) + - `mtls` - Mutual TLS or X.509 certificate validation + - Any other value will POST the client id and secret in the request body + - If not specified, defaults to `basic`. + - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. + If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. + - `client_options` are the OpenID Connect client-specific options. Specifically: + - `identifier` is the client identifier as configured in the OpenID Connect service provider. + - `secret` is the client secret as configured in the OpenID Connect service provider. + - `redirect_uri` is the GitLab URL to redirect the user to after successful login. For example, `http://example.com/users/auth/openid_connect/callback`. + - `end_session_endpoint` (optional) is the URL to the endpoint that end the session (logout). Can be provided if auto-discovery disabled or unsuccessful. + - The following `client_options` are optional unless auto-discovery is disabled or unsuccessful: + - `authorization_endpoint` is the URL to the endpoint that authorizes the end user. + - `token_endpoint` is the URL to the endpoint that provides Access Token. + - `userinfo_endpoint` is the URL to the endpoint that provides the user information. + - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. + +1. Save the configuration file. +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page, there should now be an OpenID Connect icon below the regular sign in form. Click the icon to begin the authentication process. The OpenID Connect provider will ask the user to @@ -139,25 +145,25 @@ for more details: } ``` -### Troubleshooting +## Troubleshooting If you're having trouble, here are some tips: 1. Ensure `discovery` is set to `true`. Setting it to `false` requires -specifying all the URLs and keys required to make OpenID work. + specifying all the URLs and keys required to make OpenID work. 1. Check your system clock to ensure the time is synchronized properly. 1. As mentioned in [the -documentation](https://github.com/m0n9oose/omniauth_openid_connect), -make sure `issuer` corresponds to the base URL of the Discovery URL. For -example, `https://accounts.google.com` is used for the URL -`https://accounts.google.com/.well-known/openid-configuration`. + documentation](https://github.com/m0n9oose/omniauth_openid_connect), + make sure `issuer` corresponds to the base URL of the Discovery URL. For + example, `https://accounts.google.com` is used for the URL + `https://accounts.google.com/.well-known/openid-configuration`. 1. The OpenID Connect client uses HTTP Basic Authentication to send the -OAuth2 access token. For example, if you are seeing 401 errors upon -retrieving the `userinfo` endpoint, you may want to check your OpenID -Web server configuration. For example, for -[oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you -may need to [add a configuration parameter to -Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). + OAuth2 access token if `client_auth_method` is not defined or if set to `basic`. + If you are seeing 401 errors upon retrieving the `userinfo` endpoint, you may + want to check your OpenID Web server configuration. For example, for + [oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you + may need to [add a configuration parameter to + Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index aa4e1b0d2e0..5524c3ba092 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Okta SSO provider Okta is a [Single Sign-on provider](https://www.okta.com/products/single-sign-on/) that can be used to authenticate @@ -16,7 +20,7 @@ The following documentation enables Okta as a SAML provider. 1. Next, you'll need the to fill in the SAML general config. Here's an example image. -  +  1. The last part of the configuration is the feedback section where you can just say you're a customer and creating an app for internal use. @@ -24,7 +28,7 @@ The following documentation enables Okta as a SAML provider. profile. Click on the SAML 2.0 config instructions button which should look like the following: -  +  1. On the screen that comes up take note of the **Identity Provider Single Sign-On URL** which you'll use for the @@ -38,112 +42,112 @@ Now that the Okta app is configured, it's time to enable it in GitLab. ## Configure GitLab -1. On your GitLab server, open the configuration file: - - **For Omnibus GitLab installations** - - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` - - **For installations from source** - - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` - -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) - for initial settings. - -1. To allow your users to use Okta to sign up without having to manually create - an account first, don't forget to add the following values to your - configuration: - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] - gitlab_rails['omniauth_block_auto_created_users'] = false - ``` - - **For installations from source** - - ```yaml - allow_single_sign_on: ["saml"] - block_auto_created_users: false - ``` - -1. You can also automatically link Okta users with existing GitLab users if - their email addresses match by adding the following setting: - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_auto_link_saml_user'] = true - ``` - - **For installations from source** - - ```yaml - auto_link_saml_user: true - ``` - -1. Add the provider configuration. - - >**Notes:** - > - >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint - of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab - installation to generate the correct value). - > - >- To get the `idp_cert_fingerprint` fingerprint, first download the - certificate from the Okta app you registered and then run: - `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` - with the location of your certificate. - > - >- Change the value of `idp_sso_target_url`, with the value of the - **Identity Provider Single Sign-On URL** from the step when you - configured the Okta app. - > - >- Change the value of `issuer` to the value of the **Audience Restriction** from your Okta app configuration. This will identify GitLab - to the IdP. - > - >- Leave `name_identifier_format` as-is. - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - }, - label: 'Okta' # optional label for SAML login button, defaults to "Saml" - } - ] - ``` - - **For installations from source** - - ```yaml - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - }, - label: 'Okta' # optional label for SAML login button, defaults to "Saml" - } - ``` +1. On your GitLab server, open the configuration file: + + **For Omnibus GitLab installations** + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + **For installations from source** + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) + for initial settings. + +1. To allow your users to use Okta to sign up without having to manually create + an account first, don't forget to add the following values to your + configuration: + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] + gitlab_rails['omniauth_block_auto_created_users'] = false + ``` + + **For installations from source** + + ```yaml + allow_single_sign_on: ["saml"] + block_auto_created_users: false + ``` + +1. You can also automatically link Okta users with existing GitLab users if + their email addresses match by adding the following setting: + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_auto_link_saml_user'] = true + ``` + + **For installations from source** + + ```yaml + auto_link_saml_user: true + ``` + +1. Add the provider configuration. + + >**Notes:** + > + >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint + of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab + installation to generate the correct value). + > + >- To get the `idp_cert_fingerprint` fingerprint, first download the + certificate from the Okta app you registered and then run: + `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` + with the location of your certificate. + > + >- Change the value of `idp_sso_target_url`, with the value of the + **Identity Provider Single Sign-On URL** from the step when you + configured the Okta app. + > + >- Change the value of `issuer` to the value of the **Audience Restriction** from your Okta app configuration. This will identify GitLab + to the IdP. + > + >- Leave `name_identifier_format` as-is. + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + }, + label: 'Okta' # optional label for SAML login button, defaults to "Saml" + } + ] + ``` + + **For installations from source** + + ```yaml + - { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + }, + label: 'Okta' # optional label for SAML login button, defaults to "Saml" + } + ``` 1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../restart_gitlab.md#installations-from-source) GitLab for Omnibus and installations from source respectively for the changes to take effect. @@ -157,3 +161,15 @@ Make sure the groups exist and are assigned to the Okta app. You can take a look of the [SAML documentation](../../integration/saml.md#marking-users-as-external-based-on-saml-groups) on external groups since it works the same. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index b33c5359b44..4f236d1afb8 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -1,4 +1,8 @@ -# Smartcard authentication **[PREMIUM ONLY]** +--- +type: reference +--- + +# Smartcard authentication **(PREMIUM ONLY)** GitLab supports authentication using smartcards. @@ -22,7 +26,7 @@ To use a smartcard with an X.509 certificate to authenticate against a local database with GitLab, `CN` and `emailAddress` must be defined in the certificate. For example: -``` +```text Certificate: Data: Version: 1 (0x0) @@ -56,11 +60,11 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['smartcard_enabled'] = true - gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" - gitlab_rails['smartcard_client_certificate_required_port'] = 3444 - ``` + ```ruby + gitlab_rails['smartcard_enabled'] = true + gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" + gitlab_rails['smartcard_client_certificate_required_port'] = 3444 + ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -154,15 +158,15 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - # snip... - # Enable smartcard authentication against the LDAP server. Valid values - # are "false", "optional", and "required". - smartcard_auth: optional - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + EOS + ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -171,16 +175,56 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Edit `config/gitlab.yml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - # Enable smartcard authentication against the LDAP server. Valid values - # are "false", "optional", and "required". - smartcard_auth: optional - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + ``` 1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. + +### Require browser session with smartcard sign-in for Git access + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['smartcard_required_for_git_access'] = true + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + ## Smartcard authentication settings + smartcard: + # snip... + # Browser session with smartcard sign-in is required for Git access + required_for_git_access: true + ``` + +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index 4d55f2357c1..d0adeb89543 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -1,7 +1,5 @@ # GitLab Container Registry administration -> **Notes:** -> > - [Introduced][ce-4040] in GitLab 8.8. > - Container Registry manifest `v1` support was added in GitLab 8.9 to support > Docker versions earlier than 1.10. @@ -29,8 +27,6 @@ but not recommended and out of the scope of this document. Read the [insecure Registry documentation][docker-insecure] if you want to implement this. ---- - **Installations from source** If you have installed GitLab from source: @@ -118,50 +114,44 @@ GitLab from source respectively. Be careful to choose a port different than the one that Registry listens to (`5000` by default), otherwise you will run into conflicts. ---- - **Omnibus GitLab installations** 1. Your `/etc/gitlab/gitlab.rb` should contain the Registry URL as well as the path to the existing TLS certificate and key used by GitLab: - ```ruby - registry_external_url 'https://gitlab.example.com:4567' - ``` + ```ruby + registry_external_url 'https://gitlab.example.com:4567' + ``` - Note how the `registry_external_url` is listening on HTTPS under the - existing GitLab URL, but on a different port. + Note how the `registry_external_url` is listening on HTTPS under the + existing GitLab URL, but on a different port. - If your TLS certificate is not in `/etc/gitlab/ssl/gitlab.example.com.crt` - and key not in `/etc/gitlab/ssl/gitlab.example.com.key` uncomment the lines - below: + If your TLS certificate is not in `/etc/gitlab/ssl/gitlab.example.com.crt` + and key not in `/etc/gitlab/ssl/gitlab.example.com.key` uncomment the lines + below: - ```ruby - registry_nginx['ssl_certificate'] = "/path/to/certificate.pem" - registry_nginx['ssl_certificate_key'] = "/path/to/certificate.key" - ``` + ```ruby + registry_nginx['ssl_certificate'] = "/path/to/certificate.pem" + registry_nginx['ssl_certificate_key'] = "/path/to/certificate.key" + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and configure it with the following settings: - ``` - registry: - enabled: true - host: gitlab.example.com - port: 4567 - ``` + ``` + registry: + enabled: true + host: gitlab.example.com + port: 4567 + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). ---- - Users should now be able to login to the Container Registry with their GitLab credentials using: @@ -179,8 +169,6 @@ domain (e.g., `registry.gitlab.example.com`). Let's assume that you want the container Registry to be accessible at `https://registry.gitlab.example.com`. ---- - **Omnibus GitLab installations** 1. Place your TLS certificate and key in @@ -188,17 +176,17 @@ Let's assume that you want the container Registry to be accessible at `/etc/gitlab/ssl/registry.gitlab.example.com.key` and make sure they have correct permissions: - ```bash - chmod 600 /etc/gitlab/ssl/registry.gitlab.example.com.* - ``` + ```bash + chmod 600 /etc/gitlab/ssl/registry.gitlab.example.com.* + ``` 1. Once the TLS certificate is in place, edit `/etc/gitlab/gitlab.rb` with: - ```ruby - registry_external_url 'https://registry.gitlab.example.com' - ``` + ```ruby + registry_external_url 'https://registry.gitlab.example.com' + ``` - Note how the `registry_external_url` is listening on HTTPS. + Note how the `registry_external_url` is listening on HTTPS. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. @@ -212,24 +200,20 @@ look like: > registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/certificate.key" > ``` ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and configure it with the following settings: - ``` - registry: - enabled: true - host: registry.gitlab.example.com - ``` + ```yaml + registry: + enabled: true + host: registry.gitlab.example.com + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). ---- - Users should now be able to login to the Container Registry using their GitLab credentials: @@ -248,23 +232,21 @@ Registry application itself. 1. Open `/etc/gitlab/gitlab.rb` and set `registry['enable']` to `false`: - ```ruby - registry['enable'] = false - ``` + ```ruby + registry['enable'] = false + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and set `enabled` to `false`: - ``` - registry: - enabled: false - ``` + ```yaml + registry: + enabled: false + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -274,35 +256,31 @@ If the Container Registry is enabled, then it will be available on all new projects. To disable this function and let the owners of a project to enable the Container Registry by themselves, follow the steps below. ---- - **Omnibus GitLab installations** 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['gitlab_default_projects_features_container_registry'] = false - ``` + ```ruby + gitlab_rails['gitlab_default_projects_features_container_registry'] = false + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `default_projects_features` entry and configure it so that `container_registry` is set to `false`: - ``` - ## Default project features settings - default_projects_features: - issues: true - merge_requests: true - wiki: true - snippets: false - builds: true - container_registry: false - ``` + ```yaml + ## Default project features settings + default_projects_features: + issues: true + merge_requests: true + wiki: true + snippets: false + builds: true + container_registry: false + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -323,8 +301,6 @@ This path is accessible to: > **Warning** You should confirm that all GitLab, Registry and web server users have access to this directory. ---- - **Omnibus GitLab installations** The default location where images are stored in Omnibus, is @@ -332,14 +308,12 @@ The default location where images are stored in Omnibus, is 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['registry_path'] = "/path/to/registry/storage" - ``` + ```ruby + gitlab_rails['registry_path'] = "/path/to/registry/storage" + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** The default location where images are stored in source installations, is @@ -348,10 +322,10 @@ The default location where images are stored in source installations, is 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and change the `path` setting: - ``` - registry: - path: shared/registry - ``` + ```yaml + registry: + path: shared/registry + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -393,17 +367,17 @@ To configure the `s3` storage driver in Omnibus: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - registry['storage'] = { - 's3' => { - 'accesskey' => 's3-access-key', - 'secretkey' => 's3-secret-key-for-access-key', - 'bucket' => 'your-s3-bucket', - 'region' => 'your-s3-region', - 'regionendpoint' => 'your-s3-regionendpoint' - } - } - ``` + ```ruby + registry['storage'] = { + 's3' => { + 'accesskey' => 's3-access-key', + 'secretkey' => 's3-secret-key-for-access-key', + 'bucket' => 'your-s3-bucket', + 'region' => 'your-s3-region', + 'regionendpoint' => 'your-s3-regionendpoint' + } + } + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. @@ -442,23 +416,21 @@ In the examples below we set the Registry's port to `5001`. 1. Open `/etc/gitlab/gitlab.rb` and set `registry['registry_http_addr']`: - ```ruby - registry['registry_http_addr'] = "localhost:5001" - ``` + ```ruby + registry['registry_http_addr'] = "localhost:5001" + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open the configuration file of your Registry server and edit the [`http:addr`][registry-http-config] value: - ``` - http - addr: localhost:5001 - ``` + ```yaml + http + addr: localhost:5001 + ``` 1. Save the file and restart the Registry server. @@ -476,14 +448,14 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. 1. Open `/etc/gitlab/gitlab.rb` and set necessary configurations: - ```ruby - gitlab_rails['registry_enabled'] = true - gitlab_rails['registry_host'] = "registry.gitlab.example.com" - gitlab_rails['registry_port'] = "5005" - gitlab_rails['registry_api_url'] = "http://localhost:5000" - gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry" - gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer" - ``` + ```ruby + gitlab_rails['registry_enabled'] = true + gitlab_rails['registry_host'] = "registry.gitlab.example.com" + gitlab_rails['registry_port'] = "5005" + gitlab_rails['registry_api_url'] = "http://localhost:5000" + gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry" + gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer" + ``` 1. A certificate keypair is required for GitLab and the Container Registry to communicate securely. By default omnibus-gitlab will generate one keypair, @@ -492,19 +464,19 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. custom certificate key. To do that, add the following to `/etc/gitlab/gitlab.rb` - ```ruby - gitlab_rails['registry_key_path'] = "/custom/path/to/registry-key.key" - # registry['internal_key'] should contain the contents of the custom key - # file. Line breaks in the key file should be marked using `\n` character - # Example: - registry['internal_key'] = "---BEGIN RSA PRIVATE KEY---\nMIIEpQIBAA\n" - ``` + ```ruby + gitlab_rails['registry_key_path'] = "/custom/path/to/registry-key.key" + # registry['internal_key'] should contain the contents of the custom key + # file. Line breaks in the key file should be marked using `\n` character + # Example: + registry['internal_key'] = "---BEGIN RSA PRIVATE KEY---\nMIIEpQIBAA\n" + ``` - **Note:** The file specified at `registry_key_path` gets populated with the - content specified by `internal_key`, each time reconfigure is executed. If - no file is specified, omnibus-gitlab will default it to - `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate - it. + **Note:** The file specified at `registry_key_path` gets populated with the + content specified by `internal_key`, each time reconfigure is executed. If + no file is specified, omnibus-gitlab will default it to + `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate + it. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. @@ -512,18 +484,18 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. 1. Open `/home/git/gitlab/config/gitlab.yml`, and edit the configuration settings under `registry`: - ``` - ## Container Registry + ```yaml + ## Container Registry - registry: - enabled: true - host: "registry.gitlab.example.com" - port: "5005" - api_url: "http://localhost:5000" - path: /var/opt/gitlab/gitlab-rails/shared/registry - key: /var/opt/gitlab/gitlab-rails/certificate.key - issuer: omnibus-gitlab-issuer - ``` + registry: + enabled: true + host: "registry.gitlab.example.com" + port: "5005" + api_url: "http://localhost:5000" + path: /var/opt/gitlab/gitlab-rails/shared/registry + key: /var/opt/gitlab/gitlab-rails/certificate.key + issuer: omnibus-gitlab-issuer + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -550,25 +522,23 @@ To configure a notification endpoint in Omnibus: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - registry['notifications'] = [ - { - 'name' => 'test_endpoint', - 'url' => 'https://gitlab.example.com/notify', - 'timeout' => '500ms', - 'threshold' => 5, - 'backoff' => '1s', - 'headers' => { - "Authorization" => ["AUTHORIZATION_EXAMPLE_TOKEN"] - } - } - ] - ``` + ```ruby + registry['notifications'] = [ + { + 'name' => 'test_endpoint', + 'url' => 'https://gitlab.example.com/notify', + 'timeout' => '500ms', + 'threshold' => 5, + 'backoff' => '1s', + 'headers' => { + "Authorization" => ["AUTHORIZATION_EXAMPLE_TOKEN"] + } + } + ] + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** Configuring the notification endpoint is done in your registry config YML file created @@ -629,33 +599,31 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - registry['storage'] = { - 's3' => { - 'accesskey' => 'AKIAKIAKI', - 'secretkey' => 'secret123', - 'bucket' => 'gitlab-registry-bucket-AKIAKIAKI', - 'chunksize' => 25000000 - } - } - ``` + ```ruby + registry['storage'] = { + 's3' => { + 'accesskey' => 'AKIAKIAKI', + 'secretkey' => 'secret123', + 'bucket' => 'gitlab-registry-bucket-AKIAKIAKI', + 'chunksize' => 25000000 + } + } + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **For installations from source** 1. Edit `config/gitlab.yml`: - ```yaml - storage: - s3: - accesskey: 'AKIAKIAKI' - secretkey: 'secret123' - bucket: 'gitlab-registry-bucket-AKIAKIAKI' - chunksize: 25000000 - ``` + ```yaml + storage: + s3: + accesskey: 'AKIAKIAKI' + secretkey: 'secret123' + bucket: 'gitlab-registry-bucket-AKIAKIAKI' + chunksize: 25000000 + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -669,26 +637,70 @@ You can add a configuration option for backwards compatibility. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - registry['compatibility_schema1_enabled'] = true - ``` + ```ruby + registry['compatibility_schema1_enabled'] = true + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **For installations from source** 1. Edit the YML configuration file you created when you [deployed the registry][registry-deploy]. Add the following snippet: - ```yaml - compatibility: - schema1: - enabled: true - ``` + ```yaml + compatibility: + schema1: + enabled: true + ``` 1. Restart the registry for the changes to take affect. +### Docker connection error + +A Docker connection error can occur when there are special characters in either the group, +project or branch name. Special characters can include: + +- Leading underscore +- Trailing hyphen/dash +- Double hyphen/dash + +To get around this, you can [change the group path](../user/group/index.md#changing-a-groups-path), +[change the project path](../user/project/settings/index.md#renaming-a-repository) or change the +branch name. Another option is to create a [push rule](../push_rules/push_rules.html) to prevent +this at the instance level. + +### Image push errors + +When getting errors or "retrying" loops in an attempt to push an image but `docker login` works fine, +there is likely an issue with the headers forwarded to the registry by NGINX. The default recommended +NGINX configurations should handle this, but it might occur in custom setups where the SSL is +offloaded to a third party reverse proxy. + +This problem was discussed in a [docker project issue][docker-image-push-issue] and a simple solution +would be to enable relative urls in the registry. + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['env'] = { + "REGISTRY_HTTP_RELATIVEURLS" => true + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**For installations from source** + +1. Edit the YML configuration file you created when you [deployed the registry][registry-deploy]. Add the following snippet: + + ```yaml + http: + relativeurls: true + ``` + +1. Restart the registry for the changes to take affect. [ce-18239]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18239 [docker-insecure-self-signed]: https://docs.docker.com/registry/insecure/#use-self-signed-certificates @@ -708,3 +720,4 @@ You can add a configuration option for backwards compatibility. [new-domain]: #configure-container-registry-under-its-own-domain [notifications-config]: https://docs.docker.com/registry/notifications/ [registry-notifications-config]: https://docs.docker.com/registry/configuration/#notifications +[docker-image-push-issue]: https://github.com/docker/distribution/issues/970 diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 113514e1ee8..32462a95a1a 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -21,7 +21,7 @@ administrators can add custom git hooks to any GitLab project. ## Create a custom Git hook for a repository Server-side Git hooks are typically placed in the repository's `hooks` -subdirectory. In GitLab, hook directories are are symlinked to the gitlab-shell +subdirectory. In GitLab, hook directories are symlinked to the gitlab-shell `hooks` directory for ease of maintenance between gitlab-shell upgrades. Custom hooks are implemented differently, but the behavior is exactly the same once the hook is created. Follow the steps below to set up a custom hook for a diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 7f3be402b84..dc4cc401fca 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -1,4 +1,4 @@ -# Database Load Balancing **[PREMIUM ONLY]** +# Database Load Balancing **(PREMIUM ONLY)** > [Introduced][ee-1283] in [GitLab Premium][eep] 9.0. @@ -40,16 +40,16 @@ For example, say you have a primary (`db1.gitlab.com`) and two secondaries, `db2.gitlab.com` and `db3.gitlab.com`. For this setup you will need to have 3 load balancers, one for every host. For example: -* `primary.gitlab.com` forwards to `db1.gitlab.com` -* `secondary1.gitlab.com` forwards to `db2.gitlab.com` -* `secondary2.gitlab.com` forwards to `db3.gitlab.com` +- `primary.gitlab.com` forwards to `db1.gitlab.com` +- `secondary1.gitlab.com` forwards to `db2.gitlab.com` +- `secondary2.gitlab.com` forwards to `db3.gitlab.com` Now let's say that a failover happens and db2 becomes the new primary. This means forwarding should now happen as follows: -* `primary.gitlab.com` forwards to `db2.gitlab.com` -* `secondary1.gitlab.com` forwards to `db1.gitlab.com` -* `secondary2.gitlab.com` forwards to `db3.gitlab.com` +- `primary.gitlab.com` forwards to `db2.gitlab.com` +- `secondary1.gitlab.com` forwards to `db1.gitlab.com` +- `secondary2.gitlab.com` forwards to `db3.gitlab.com` GitLab does not take care of this for you, so you will need to do so yourself. @@ -74,9 +74,9 @@ the following. This will balance the load between `host1.example.com` and 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } - ``` + ```ruby + gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. @@ -86,16 +86,16 @@ the following. This will balance the load between `host1.example.com` and 1. Edit `/home/git/gitlab/config/database.yml` and add or amend the following lines: - ```yaml - production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - hosts: - - host1.example.com - - host2.example.com - ``` + ```yaml + production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + hosts: + - host1.example.com + - host2.example.com + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -209,9 +209,9 @@ without it immediately leading to errors being presented to the users. The load balancer logs various messages, such as: -* When a host is marked as offline -* When a host comes back online -* When all secondaries are offline +- When a host is marked as offline +- When a host comes back online +- When all secondaries are offline Each log message contains the tag `[DB-LB]` to make searching/filtering of such log entries easier. For example: @@ -265,7 +265,7 @@ production: replica_check_interval: 30 ``` -[hot-standby]: https://www.postgresql.org/docs/9.6/static/hot-standby.html +[hot-standby]: https://www.postgresql.org/docs/9.6/hot-standby.html [ee-1283]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1283 [eep]: https://about.gitlab.com/pricing/ [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" diff --git a/doc/administration/dependency_proxy.md b/doc/administration/dependency_proxy.md index 4dc1f4dcba4..d2c52b67e67 100644 --- a/doc/administration/dependency_proxy.md +++ b/doc/administration/dependency_proxy.md @@ -1,6 +1,6 @@ -# GitLab Dependency Proxy administration **[PREMIUM ONLY]** +# GitLab Dependency Proxy administration **(PREMIUM ONLY)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing) 11.11. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. GitLab can be utilized as a dependency proxy for a variety of common package managers. @@ -70,6 +70,7 @@ To change the local storage path: enabled: true storage_path: shared/dependency_proxy ``` + 1. [Restart GitLab] for the changes to take effect. ### Using object storage diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index e6c8f59549f..37d7194af53 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -13,6 +13,7 @@ override certain values. Variable | Type | Description -------- | ---- | ----------- +`ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable) `GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (e.g. `//mycdnsubdomain.fictional-cdn.com`) `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) @@ -42,7 +43,7 @@ The list of `GITLAB_DATABASE_XXX` variables that you can set is: Variable | Default value | Overridden by `DATABASE_URL`? -------- | ------------- | ----------------------------- -`GITLAB_DATABASE_ADAPTER` | `postgresql` (for MySQL use `mysql2`) | Yes +`GITLAB_DATABASE_ADAPTER` | `postgresql` | Yes `GITLAB_DATABASE_DATABASE` | `gitlab_#{ENV['RAILS_ENV']` | Yes `GITLAB_DATABASE_USERNAME` | `root` | Yes `GITLAB_DATABASE_PASSWORD` | None | Yes @@ -64,4 +65,4 @@ instructions](https://docs.gitlab.com/omnibus/settings/environment-variables.htm It's possible to preconfigure the GitLab docker image by adding the environment variable `GITLAB_OMNIBUS_CONFIG` to the `docker run` command. -For more information see the ['preconfigure-docker-container' section in the Omnibus documentation](http://docs.gitlab.com/omnibus/docker/#preconfigure-docker-container). +For more information see the ['preconfigure-docker-container' section in the Omnibus documentation](https://docs.gitlab.com/omnibus/docker/#preconfigure-docker-container). diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index d4c8c2d3624..27866b7536e 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -1,4 +1,4 @@ -# Automatic background verification **[PREMIUM ONLY]** +# Automatic background verification **(PREMIUM ONLY)** NOTE: **Note:** Automatic background verification of repositories and wikis was added in @@ -143,11 +143,13 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch 1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:  -1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure. +1. Navigate to the project's repository directory on both **primary** and **secondary** nodes + (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` + is customized, check the directory layout on your server to be sure. - ```sh - cd /var/opt/gitlab/git-data/repositories - ``` + ```sh + cd /var/opt/gitlab/git-data/repositories + ``` 1. Run the following command on the **primary** node, redirecting the output to a file: @@ -169,14 +171,21 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch ## Current limitations -Until [issue #5064][ee-5064] is completed, background verification doesn't cover -CI job artifacts and traces, LFS objects, or user uploads in file storage. -Verify their integrity manually by following [these instructions][foreground-verification] -on both nodes, and comparing the output between them. +Automatic background verification doesn't cover attachments, LFS objects, +job artifacts, and user uploads in file storage. You can keep track of the +progress to include them in [ee-1430]. For now, you can verify their integrity +manually by following [these instructions][foreground-verification] on both +nodes, and comparing the output between them. + +In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects and +archived traces on secondary nodes after the transfer, compares it with the +stored checksums, and rejects transfers if mismatched. Please note that Geo +currently does not support an automatic way to verify these data if they have +been synced before GitLab EE 12.1. Data in object storage is **not verified**, as the object store is responsible for ensuring the integrity of the data. [reset-verification]: background_verification.md#reset-verification-for-projects-where-verification-has-failed [foreground-verification]: ../../raketasks/check.md -[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 +[ee-1430]: https://gitlab.com/groups/gitlab-org/-/epics/1430 diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index f4d31a98080..64d7ef2d609 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -1,4 +1,4 @@ -# Bring a demoted primary node back online **[PREMIUM ONLY]** +# Bring a demoted primary node back online **(PREMIUM ONLY)** After a failover, it is possible to fail back to the demoted **primary** node to restore your original configuration. This process consists of two steps: @@ -21,20 +21,20 @@ To bring the former **primary** node up to date: 1. SSH into the former **primary** node that has fallen behind. 1. Make sure all the services are up: - ```sh - sudo gitlab-ctl start - ``` - - > **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], - > you need to undo those steps now. For Debian/Ubuntu you just need to run - > `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install - > the GitLab instance from scratch and set it up as a **secondary** node by - > following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. - > - > **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) - > for this node during disaster recovery procedure you may need to [block - > all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic) - > during this procedure. + ```sh + sudo gitlab-ctl start + ``` + + NOTE: **Note:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], + you need to undo those steps now. For Debian/Ubuntu you just need to run + `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install + the GitLab instance from scratch and set it up as a **secondary** node by + following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. + + NOTE: **Note:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) + for this node during disaster recovery procedure you may need to [block + all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) + during this procedure. 1. [Setup database replication][database-replication]. Note that in this case, **primary** node refers to the current **primary** node, and **secondary** node refers to the diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png Binary files differindex fd51523104b..0a7b841897b 100644 --- a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png Binary files differindex b2a6da69d3d..85759d903a4 100644 --- a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 71dc797f281..ba95843b0b0 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,4 +1,4 @@ -# Disaster Recovery **[PREMIUM ONLY]** +# Disaster Recovery (Geo) **(PREMIUM ONLY)** Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to @@ -39,48 +39,50 @@ must disable the **primary** node. 1. SSH into the **primary** node to stop and disable GitLab, if possible: - ```sh - sudo gitlab-ctl stop - ``` + ```sh + sudo gitlab-ctl stop + ``` - Prevent GitLab from starting up again if the server unexpectedly reboots: + Prevent GitLab from starting up again if the server unexpectedly reboots: - ```sh - sudo systemctl disable gitlab-runsvdir - ``` + ```sh + sudo systemctl disable gitlab-runsvdir + ``` - > **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being - > started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). - > It may be safest to uninstall the GitLab package completely: + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). + It may be safest to uninstall the GitLab package completely: - ```sh - yum remove gitlab-ee - ``` + ```sh + yum remove gitlab-ee + ``` - > **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu - > or any other distro based on the Upstart init system, you can prevent GitLab - > from starting if the machine reboots by doing the following: + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distro based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots by doing the following: - ```sh - initctl stop gitlab-runsvvdir - echo 'manual' > /etc/init/gitlab-runsvdir.override - initctl reload-configuration - ``` + ```sh + initctl stop gitlab-runsvvdir + echo 'manual' > /etc/init/gitlab-runsvdir.override + initctl reload-configuration + ``` 1. If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting by any means at your disposal. - Since there are many ways you may prefer to accomplish this, we will avoid a - single recommendation. You may need to: - - Reconfigure the load balancers. - - Change DNS records (e.g., point the primary DNS record to the **secondary** - node in order to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. - -1. If you plan to - [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (e.g., point the primary DNS record to the **secondary** + node in order to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. ### Step 3. Promoting a **secondary** node @@ -94,26 +96,26 @@ the **secondary** to the **primary**. 1. SSH in to your **secondary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` 1. Promote the **secondary** node to the **primary** node. Execute: - ```sh - gitlab-ctl promote-to-primary-node - ``` + ```sh + gitlab-ctl promote-to-primary-node + ``` 1. Verify you can connect to the newly promoted **primary** node using the URL used previously for the **secondary** node. @@ -129,31 +131,31 @@ do this manually. 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: - ```bash - sudo gitlab-pg-ctl promote - ``` + ```bash + sudo gitlab-pg-ctl promote + ``` 1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each - machine so the changes take effect. + After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single application server and execute: - ```bash - sudo gitlab-rake geo:set_secondary_as_primary - ``` + ```bash + sudo gitlab-rake geo:set_secondary_as_primary + ``` 1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**. @@ -167,37 +169,37 @@ secondary domain, like changing Git remotes and API URLs. 1. SSH into the **secondary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Update the primary domain's DNS record. After updating the primary domain's DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the **secondary** node to reflect the new URL: - ```ruby - # Change the existing external_url configuration - external_url 'https://<new_external_url>' - ``` + ```ruby + # Change the existing external_url configuration + external_url 'https://<new_external_url>' + ``` - NOTE: **Note** - Changing `external_url` won't prevent access via the old secondary URL, as - long as the secondary DNS records are still intact. + NOTE: **Note** + Changing `external_url` won't prevent access via the old secondary URL, as + long as the secondary DNS records are still intact. 1. Reconfigure the **secondary** node for the change to take effect: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Execute the command below to update the newly promoted **primary** node URL: - ```sh - gitlab-rake geo:update_primary_node_url - ``` + ```sh + gitlab-rake geo:update_primary_node_url + ``` - This command will use the changed `external_url` configuration defined - in `/etc/gitlab/gitlab.rb`. + This command will use the changed `external_url` configuration defined + in `/etc/gitlab/gitlab.rb`. 1. Verify you can connect to the newly promoted **primary** using its URL. If you updated the DNS records for the primary domain, these changes may @@ -231,62 +233,61 @@ and after that you also need two extra steps. 1. SSH into the new **primary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` - ```ruby - ## Enable a Geo Primary role (if you haven't yet) - roles ['geo_primary_role'] + ```ruby + ## Enable a Geo Primary role (if you haven't yet) + roles ['geo_primary_role'] - ## - # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be - # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] - ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] - # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have - postgresql['max_replication_slots'] = 1 + # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have + postgresql['max_replication_slots'] = 1 - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## - gitlab_rails['auto_migrate'] = false + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` - ``` - - (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) + (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied. - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` - Restart PostgreSQL for its changes to take effect: + Restart PostgreSQL for its changes to take effect: - ```sh - gitlab-ctl restart postgresql - ``` + ```sh + gitlab-ctl restart postgresql + ``` 1. Re-enable migrations now that PostgreSQL is restarted and listening on the private address. - Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: - ```ruby - gitlab_rails['auto_migrate'] = true - ``` + ```ruby + gitlab_rails['auto_migrate'] = true + ``` - Save the file and reconfigure GitLab: + Save the file and reconfigure GitLab: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` ### Step 2. Initiate the replication process diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index b8071b5993f..0c2d48854f0 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -1,4 +1,4 @@ -# Disaster recovery for planned failover **[PREMIUM ONLY]** +# Disaster recovery for planned failover **(PREMIUM ONLY)** The primary use-case of Disaster Recovery is to ensure business continuity in the event of unplanned outage, but it can be used in conjunction with a planned @@ -143,26 +143,26 @@ access to the **primary** node during the maintenance window. all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and the **secondary** node's IP. - For instance, you might run the following commands on the server(s) making up your **primary** node: + For instance, you might run the following commands on the server(s) making up your **primary** node: - ```sh - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT --destination-port 22 -j REJECT + ```sh + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 80 -j REJECT + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 443 -j REJECT - ``` + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` - From this point, users will be unable to view their data or make changes on the - **primary** node. They will also be unable to log in to the **secondary** node. - However, existing sessions will work for the remainder of the maintenance period, and - public data will be accessible throughout. + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via another IP. The server should refuse connection. @@ -187,10 +187,11 @@ access to the **primary** node during the maintenance window. before it is completed will cause the work to be lost. 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the following conditions to be true of the **secondary** node you are failing over to: - - All replication meters to each 100% replicated, 0% failures. - - All verification meters reach 100% verified, 0% failures. - - Database replication lag is 0ms. - - The Geo log cursor is up to date (0 events behind). + + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 3d4f69d3abe..fd076bb79d8 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -1,4 +1,4 @@ -# Geo configuration **[PREMIUM ONLY]** +# Geo configuration **(PREMIUM ONLY)** ## Configuring a new **secondary** node @@ -16,11 +16,10 @@ The basic steps of configuring a **secondary** node are to: You are encouraged to first read through all the steps before executing them in your testing/production environment. -> **Notes:** -> - **Do not** setup any custom authentication for the **secondary** nodes. This will be - handled by the **primary** node. -> - Any change that requires access to the **Admin Area** needs to be done in the - **primary** node because the **secondary** node is a read-only replica. +NOTE: **Note:** +**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. +Any change that requires access to the **Admin Area** needs to be done in the +**primary** node because the **secondary** node is a read-only replica. ### Step 1. Manually replicate secret GitLab values @@ -31,47 +30,47 @@ they must be manually replicated to the **secondary** node. 1. SSH into the **primary** node, and execute the command below: - ```sh - sudo cat /etc/gitlab/gitlab-secrets.json - ``` + ```sh + sudo cat /etc/gitlab/gitlab-secrets.json + ``` - This will display the secrets that need to be replicated, in JSON format. + This will display the secrets that need to be replicated, in JSON format. 1. SSH into the **secondary** node and login as the `root` user: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Make a backup of any existing secrets: - ```sh - mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` - ``` + ```sh + mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` + ``` 1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or copy-and-paste the file contents between nodes: - ```sh - sudo editor /etc/gitlab/gitlab-secrets.json + ```sh + sudo editor /etc/gitlab/gitlab-secrets.json - # paste the output of the `cat` command you ran on the primary - # save and exit - ``` + # paste the output of the `cat` command you ran on the primary + # save and exit + ``` 1. Ensure the file permissions are correct: - ```sh - chown root:root /etc/gitlab/gitlab-secrets.json - chmod 0600 /etc/gitlab/gitlab-secrets.json - ``` + ```sh + chown root:root /etc/gitlab/gitlab-secrets.json + chmod 0600 /etc/gitlab/gitlab-secrets.json + ``` 1. Reconfigure the **secondary** node for the change to take effect: - ```sh - gitlab-ctl reconfigure - gitlab-ctl restart - ``` + ```sh + gitlab-ctl reconfigure + gitlab-ctl restart + ``` ### Step 2. Manually replicate the **primary** node's SSH host keys @@ -89,80 +88,80 @@ keys must be manually replicated to the **secondary** node. 1. SSH into the **secondary** node and login as the `root` user: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Make a backup of any existing SSH host keys: - ```sh - find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; - ``` + ```sh + find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; + ``` 1. Copy OpenSSH host keys from the **primary** node: - If you can access your **primary** node using the **root** user: + If you can access your **primary** node using the **root** user: - ```sh - # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server - scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh - ``` + ```sh + # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server + scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh + ``` - If you only have access through a user with **sudo** privileges: + If you only have access through a user with **sudo** privileges: - ```sh - # Run this from your primary node: - sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* + ```sh + # Run this from your primary node: + sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* - # Run this from your secondary node: - scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . - tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh - ``` + # Run this from your secondary node: + scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . + tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh + ``` 1. On your **secondary** node, ensure the file permissions are correct: - ```sh - chown root:root /etc/ssh/ssh_host_*_key* - chmod 0600 /etc/ssh/ssh_host_*_key* - ``` + ```sh + chown root:root /etc/ssh/ssh_host_*_key* + chmod 0600 /etc/ssh/ssh_host_*_key* + ``` 1. To verify key fingerprint matches, execute the following command on both nodes: - ```sh - for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done - ``` + ```sh + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ``` - You should get an output similar to this one and they should be identical on both nodes: + You should get an output similar to this one and they should be identical on both nodes: - ```sh - 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) - 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) - 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) - 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) - ``` + ```sh + 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) + 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) + 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) + 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) + ``` 1. Verify that you have the correct public keys for the existing private keys: - ```sh - # This will print the fingerprint for private keys: - for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ```sh + # This will print the fingerprint for private keys: + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done - # This will print the fingerprint for public keys: - for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done - ``` + # This will print the fingerprint for public keys: + for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done + ``` - NOTE: **Note**: - The output for private keys and public keys command should generate the same fingerprint. + NOTE: **Note:** + The output for private keys and public keys command should generate the same fingerprint. 1. Restart sshd on your **secondary** node: - ```sh - # Debian or Ubuntu installations - sudo service ssh reload + ```sh + # Debian or Ubuntu installations + sudo service ssh reload - # CentOS installations - sudo service sshd reload - ``` + # CentOS installations + sudo service sshd reload + ``` ### Step 3. Add the **secondary** node @@ -176,22 +175,22 @@ keys must be manually replicated to the **secondary** node. 1. Click the **Add node** button. 1. SSH into your GitLab **secondary** server and restart the services: - ```sh - gitlab-ctl restart - ``` + ```sh + gitlab-ctl restart + ``` - Check if there are any common issue with your Geo setup by running: + Check if there are any common issue with your Geo setup by running: - ```sh - gitlab-rake gitlab:geo:check - ``` + ```sh + gitlab-rake gitlab:geo:check + ``` 1. SSH into your **primary** server and login as root to verify the **secondary** node is reachable or there are any common issue with your Geo setup: - ```sh - gitlab-rake gitlab:geo:check - ``` + ```sh + gitlab-rake gitlab:geo:check + ``` Once added to the admin panel and restarted, the **secondary** node will automatically start replicating missing data from the **primary** node in a process known as **backfill**. @@ -243,16 +242,15 @@ node's Geo Nodes dashboard in your browser.  If your installation isn't working properly, check the -[troubleshooting document]. +[troubleshooting document](troubleshooting.md). The two most obvious issues that can become apparent in the dashboard are: 1. Database replication not working well. 1. Instance to instance notification not working. In that case, it can be something of the following: - - You are using a custom certificate or custom CA (see the - [troubleshooting document]). - - The instance is firewalled (check your firewall rules). + - You are using a custom certificate or custom CA (see the [troubleshooting document](troubleshooting.md)). + - The instance is firewalled (check your firewall rules). Please note that disabling a **secondary** node will stop the synchronization process. @@ -304,5 +302,4 @@ See the [troubleshooting document](troubleshooting.md). [gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789 [gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 [omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html -[troubleshooting document]: troubleshooting.md [using-geo]: using_a_geo_server.md diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 1e5a56c3f4e..9272287a4a7 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -1,4 +1,4 @@ -# Geo database replication **[PREMIUM ONLY]** +# Geo database replication **(PREMIUM ONLY)** NOTE: **Note:** The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5. @@ -52,186 +52,188 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. SSH into your GitLab **primary** server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Execute the command below to define the node as **primary** node: - ```sh - gitlab-ctl set-geo-primary-node - ``` + ```sh + gitlab-ctl set-geo-primary-node + ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. 1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: - Generate a MD5 hash of the desired password: + Generate a MD5 hash of the desired password: - ```sh - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` - Edit `/etc/gitlab/gitlab.rb`: + Edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` 1. Omnibus GitLab already has a [replication user] called `gitlab_replicator`. You must set the password for this user manually. You will be prompted to enter a password: - ```sh - gitlab-ctl set-replication-password - ``` + ```sh + gitlab-ctl set-replication-password + ``` - This command will also read the `postgresql['sql_replication_user']` Omnibus - setting in case you have changed `gitlab_replicator` username to something - else. + This command will also read the `postgresql['sql_replication_user']` Omnibus + setting in case you have changed `gitlab_replicator` username to something + else. - If you are using an external database not managed by Omnibus GitLab, you need - to create the replicator user and define a password to it manually: + If you are using an external database not managed by Omnibus GitLab, you need + to create the replicator user and define a password to it manually: - ```sql - --- Create a new user 'replicator' - CREATE USER gitlab_replicator; + ```sql + --- Create a new user 'replicator' + CREATE USER gitlab_replicator; - --- Set/change a password and grants replication privilege - ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; - ``` + --- Set/change a password and grants replication privilege + ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; + ``` 1. Configure PostgreSQL to listen on network interfaces: - For security reasons, PostgreSQL does not listen on any network interfaces - by default. However, Geo requires the **secondary** node to be able to - connect to the **primary** node's database. For this reason, we need the address of - each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md). - - If you are using a cloud provider, you can lookup the addresses for each - Geo node through your cloud provider's management console. - - To lookup the address of a Geo node, SSH in to the Geo node and execute: - - ```sh - ## - ## Private address - ## - ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' - - ## - ## Public address - ## - echo "External address: $(curl --silent ipinfo.io/ip)" - ``` - - In most cases, the following addresses will be used to configure GitLab - Geo: - - | Configuration | Address | - |:----------------------------------------|:------------------------------------------------------| - | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | - | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | - - If you are using Google Cloud Platform, SoftLayer, or any other vendor that - provides a virtual private cloud (VPC) you can use the **secondary** node's private - address (corresponds to "internal address" for Google Cloud Platform) for - `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. - - The `listen_address` option opens PostgreSQL up to network connections - with the interface corresponding to the given address. See [the PostgreSQL - documentation][pg-docs-runtime-conn] for more details. - - Depending on your network configuration, the suggested addresses may not - be correct. If your **primary** node and **secondary** nodes connect over a local - area network, or a virtual network connecting availability zones like - [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) - you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. - - Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP - addresses with addresses appropriate to your network configuration: - - ```ruby - ## - ## Geo Primary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_primary_role'] - - ## - ## Primary address - ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node - ## - postgresql['listen_address'] = '<primary_node_ip>' - - ## - # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be - # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] - ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] - - ## - ## Replication settings - ## - set this to be the number of Geo secondary nodes you have - ## - postgresql['max_replication_slots'] = 1 - # postgresql['max_wal_senders'] = 10 - # postgresql['wal_keep_segments'] = 10 - - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## - gitlab_rails['auto_migrate'] = false - ``` + For security reasons, PostgreSQL does not listen on any network interfaces + by default. However, Geo requires the **secondary** node to be able to + connect to the **primary** node's database. For this reason, we need the address of + each node. + + NOTE: **Note:** For external PostgreSQL instances, see [additional instructions](external_database.md). + + If you are using a cloud provider, you can lookup the addresses for each + Geo node through your cloud provider's management console. + + To lookup the address of a Geo node, SSH in to the Geo node and execute: + + ```sh + ## + ## Private address + ## + ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' + + ## + ## Public address + ## + echo "External address: $(curl --silent ipinfo.io/ip)" + ``` + + In most cases, the following addresses will be used to configure GitLab + Geo: + + | Configuration | Address | + |:----------------------------------------|:------------------------------------------------------| + | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | + + If you are using Google Cloud Platform, SoftLayer, or any other vendor that + provides a virtual private cloud (VPC) you can use the **secondary** node's private + address (corresponds to "internal address" for Google Cloud Platform) for + `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. + + The `listen_address` option opens PostgreSQL up to network connections + with the interface corresponding to the given address. See [the PostgreSQL + documentation][pg-docs-runtime-conn] for more details. + + Depending on your network configuration, the suggested addresses may not + be correct. If your **primary** node and **secondary** nodes connect over a local + area network, or a virtual network connecting availability zones like + [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) + you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## Primary address + ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node + ## + postgresql['listen_address'] = '<primary_node_ip>' + + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + + ## + ## Replication settings + ## - set this to be the number of Geo secondary nodes you have + ## + postgresql['max_replication_slots'] = 1 + # postgresql['max_wal_senders'] = 10 + # postgresql['wal_keep_segments'] = 10 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` 1. Optional: If you want to add another **secondary** node, the relevant setting would look like: - ```ruby - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] - ``` + ```ruby + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] + ``` - You may also want to edit the `wal_keep_segments` and `max_wal_senders` to - match your database replication requirements. Consult the [PostgreSQL - - Replication documentation][pg-docs-runtime-replication] - for more information. + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to + match your database replication requirements. Consult the [PostgreSQL - + Replication documentation][pg-docs-runtime-replication] + for more information. 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` - Restart PostgreSQL for its changes to take effect: + Restart PostgreSQL for its changes to take effect: - ```sh - gitlab-ctl restart postgresql - ``` + ```sh + gitlab-ctl restart postgresql + ``` 1. Re-enable migrations now that PostgreSQL is restarted and listening on the private address. - Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: - ```ruby - gitlab_rails['auto_migrate'] = true - ``` + ```ruby + gitlab_rails['auto_migrate'] = true + ``` - Save the file and reconfigure GitLab: + Save the file and reconfigure GitLab: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Now that the PostgreSQL server is set up to accept remote connections, run `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port @@ -241,143 +243,143 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o will be used automatically to protect your PostgreSQL traffic from eavesdroppers, but to protect against active ("man-in-the-middle") attackers, the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL - `server.crt` file on the **primary** node by running this command: + `server.crt` file on the **primary** node by running this command: - ```sh - cat ~gitlab-psql/data/server.crt - ``` + ```sh + cat ~gitlab-psql/data/server.crt + ``` - Copy the output into a clipboard or into a local file. You - will need it when setting up the **secondary** node! The certificate is not sensitive - data. + Copy the output into a clipboard or into a local file. You + will need it when setting up the **secondary** node! The certificate is not sensitive + data. ### Step 2. Configure the **secondary** server 1. SSH into your GitLab **secondary** server and login as root: - ``` - sudo -i - ``` + ``` + sudo -i + ``` 1. Stop application server and Sidekiq - ``` - gitlab-ctl stop unicorn - gitlab-ctl stop sidekiq - ``` + ``` + gitlab-ctl stop unicorn + gitlab-ctl stop sidekiq + ``` - NOTE: **Note**: - This step is important so we don't try to execute anything before the node is fully configured. + NOTE: **Note:** + This step is important so we don't try to execute anything before the node is fully configured. 1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server: - ```sh - gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] - ``` + ```sh + gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] + ``` - NOTE: **Note**: - If this step fails, you may be using the wrong IP address, or a firewall may - be preventing access to the server. Check the IP address, paying close - attention to the difference between public and private addresses and ensure - that, if a firewall is present, the **secondary** node is permitted to connect to the - **primary** node on port 5432. + NOTE: **Note:** + If this step fails, you may be using the wrong IP address, or a firewall may + be preventing access to the server. Check the IP address, paying close + attention to the difference between public and private addresses and ensure + that, if a firewall is present, the **secondary** node is permitted to connect to the + **primary** node on port 5432. 1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: - ``` - editor server.crt - ``` + ``` + editor server.crt + ``` 1. Set up PostgreSQL TLS verification on the **secondary** node: - Install the `server.crt` file: + Install the `server.crt` file: - ```sh - install \ - -D \ - -o gitlab-psql \ - -g gitlab-psql \ - -m 0400 \ - -T server.crt ~gitlab-psql/.postgresql/root.crt - ``` + ```sh + install \ + -D \ + -o gitlab-psql \ + -g gitlab-psql \ + -m 0400 \ + -T server.crt ~gitlab-psql/.postgresql/root.crt + ``` - PostgreSQL will now only recognize that exact certificate when verifying TLS - connections. The certificate can only be replicated by someone with access - to the private key, which is **only** present on the **primary** node. + PostgreSQL will now only recognize that exact certificate when verifying TLS + connections. The certificate can only be replicated by someone with access + to the private key, which is **only** present on the **primary** node. 1. Test that the `gitlab-psql` user can connect to the **primary** node's database (the default Omnibus database name is gitlabhq_production): - ```sh - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - --list \ - -U gitlab_replicator \ - -d "dbname=gitlabhq_production sslmode=verify-ca" \ - -W \ - -h <primary_node_ip> - ``` + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + --list \ + -U gitlab_replicator \ + -d "dbname=gitlabhq_production sslmode=verify-ca" \ + -W \ + -h <primary_node_ip> + ``` - When prompted enter the password you set in the first step for the - `gitlab_replicator` user. If all worked correctly, you should see - the list of **primary** node's databases. + When prompted enter the password you set in the first step for the + `gitlab_replicator` user. If all worked correctly, you should see + the list of **primary** node's databases. - A failure to connect here indicates that the TLS configuration is incorrect. - Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node - match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + A failure to connect here indicates that the TLS configuration is incorrect. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. 1. Configure PostgreSQL to enable FDW support: - This step is similar to how we configured the **primary** instance. - We need to enable this, to enable FDW support, even if using a single node. - - Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP - addresses with addresses appropriate to your network configuration: - - ```ruby - ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_secondary_role'] - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## Enable FDW support for the Geo Tracking Database (improves performance) - ## - geo_secondary['db_fdw'] = true - ``` - - For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + This step is similar to how we configured the **primary** instance. + We need to enable this, to enable FDW support, even if using a single node. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## Enable FDW support for the Geo Tracking Database (improves performance) + ## + geo_secondary['db_fdw'] = true + ``` + + For external PostgreSQL instances, see [additional instructions](external_database.md). + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Restart PostgreSQL for the IP change to take effect and reconfigure again: - ```sh - gitlab-ctl restart postgresql - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl restart postgresql + gitlab-ctl reconfigure + ``` - This last reconfigure will provision the FDW configuration and enable it. + This last reconfigure will provision the FDW configuration and enable it. ### Step 3. Initiate the replication process @@ -394,9 +396,9 @@ data before running `pg_basebackup`. 1. SSH into your GitLab **secondary** server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Choose a database-friendly name to use for your **secondary** node to use as the replication slot name. For example, if your domain is @@ -404,38 +406,40 @@ data before running `pg_basebackup`. name as shown in the commands below. 1. Execute the command below to start a backup/restore and begin the replication - CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. - Using the same slot name between two secondaries will break PostgreSQL replication. - - ```sh - gitlab-ctl replicate-geo-database \ - --slot-name=<secondary_node_name> \ - --host=<primary_node_ip> - ``` - - When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` - user in the first step. - - This command also takes a number of additional options. You can use `--help` - to list them all, but here are a couple of tips: - - If PostgreSQL is listening on a non-standard port, add `--port=` as well. - - If your database is too large to be transferred in 30 minutes, you will need - to increase the timeout, e.g., `--backup-timeout=3600` if you expect the - initial replication to take under an hour. - - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether - (e.g., you know the network path is secure, or you are using a site-to-site - VPN). This is **not** safe over the public Internet! - - You can read more details about each `sslmode` in the - [PostgreSQL documentation][pg-docs-ssl]; - the instructions above are carefully written to ensure protection against - both passive eavesdroppers and active "man-in-the-middle" attackers. - - Change the `--slot-name` to the name of the replication slot - to be used on the **primary** database. The script will attempt to create the - replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you'll need to - add `--force` to the command line. - - When not in a production machine you can disable backup step if you - really sure this is what you want by adding `--skip-backup` + + CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. + Using the same slot name between two secondaries will break PostgreSQL replication. + + ```sh + gitlab-ctl replicate-geo-database \ + --slot-name=<secondary_node_name> \ + --host=<primary_node_ip> + ``` + + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` + user in the first step. + + This command also takes a number of additional options. You can use `--help` + to list them all, but here are a couple of tips: + + - If PostgreSQL is listening on a non-standard port, add `--port=` as well. + - If your database is too large to be transferred in 30 minutes, you will need + to increase the timeout, e.g., `--backup-timeout=3600` if you expect the + initial replication to take under an hour. + - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether + (e.g., you know the network path is secure, or you are using a site-to-site + VPN). This is **not** safe over the public Internet! + - You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + - Change the `--slot-name` to the name of the replication slot + to be used on the **primary** database. The script will attempt to create the + replication slot automatically if it does not exist. + - If you're repurposing an old server into a Geo **secondary** node, you'll need to + add `--force` to the command line. + - When not in a production machine you can disable backup step if you + really sure this is what you want by adding `--skip-backup` The replication process is now complete. @@ -452,46 +456,42 @@ it will need a separate read-only user to make [PostgreSQL FDW queries][FDW] work: 1. On the **primary** Geo database, enter the PostgreSQL on the console as an - admin user. If you are using an Omnibus-managed database, log onto the **primary** - node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production): + admin user. If you are using an Omnibus-managed database, log onto the **primary** + node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production): - ```sh - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/postgresql gitlabhq_production - ``` + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/postgresql gitlabhq_production + ``` 1. Then create the read-only user: - ```sql - -- NOTE: Use the password defined earlier - CREATE USER gitlab_geo_fdw WITH password 'mypassword'; - GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; - GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; - GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; - GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_geo_fdw WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; + GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; - -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" - -- automatically. - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; - ``` + -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; + ``` 1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`: - ``` - geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' - ``` + ``` + geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' + ``` 1. Save the file and reconfigure GitLab for the changes to be applied: - ```sh - gitlab-ctl reconfigure - ``` - -## MySQL replication - -MySQL replication is not supported for Geo. + ```sh + gitlab-ctl reconfigure + ``` ## Troubleshooting diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 5b02b861c61..d5c2d2362b1 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -1,4 +1,4 @@ -# Docker Registry for a secondary node **[PREMIUM ONLY]** +# Docker Registry for a secondary node **(PREMIUM ONLY)** You can set up a [Docker Registry] on your **secondary** Geo node that mirrors the one on the **primary** Geo node. diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index 177ca68613e..256195998a7 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -1,10 +1,10 @@ -# Geo with external PostgreSQL instances **[PREMIUM ONLY]** +# Geo with external PostgreSQL instances **(PREMIUM ONLY)** This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or manually installed and configured PostgreSQL instances. -NOTE: **Note**: +NOTE: **Note:** We strongly recommend running Omnibus-managed instances as they are actively developed and tested. We aim to be compatible with most external (not managed by Omnibus) databases but we do not guarantee compatibility. @@ -13,18 +13,17 @@ developed and tested. We aim to be compatible with most external 1. SSH into a GitLab **primary** application server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Execute the command below to define the node as **primary** node: - ```sh - gitlab-ctl set-geo-primary-node - ``` - - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + ```sh + gitlab-ctl set-geo-primary-node + ``` + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. ### Configure the external database to be replicated @@ -101,26 +100,27 @@ To configure the connection to the external read-replica database and enable Log 1. SSH into a GitLab **secondary** application server and login as root: - ```bash - sudo -i - ``` + ```bash + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` and add the following - ```ruby - ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_secondary_role'] + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + # note this is shared between both databases, + # make sure you define the same password in both + gitlab_rails['db_password'] = '<your_password_here>' - # note this is shared between both databases, - # make sure you define the same password in both - gitlab_rails['db_password'] = '<your_password_here>' + gitlab_rails['db_username'] = 'gitlab' + gitlab_rails['db_host'] = '<database_read_replica_host>' + ``` - gitlab_rails['db_username'] = 'gitlab' - gitlab_rails['db_host'] = '<database_read_replica_host>' - ``` 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) ### Configure the tracking database @@ -147,73 +147,72 @@ the tracking database on port 5432. 1. SSH into a GitLab **secondary** server and login as root: - ```bash - sudo -i - ``` + ```bash + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials for - the machine with the PostgreSQL instance: + the machine with the PostgreSQL instance: - ```ruby - geo_secondary['db_username'] = 'gitlab_geo' - geo_secondary['db_password'] = '<your_password_here>' + ```ruby + geo_secondary['db_username'] = 'gitlab_geo' + geo_secondary['db_password'] = '<your_password_here>' - geo_secondary['db_host'] = '<tracking_database_host>' - geo_secondary['db_port'] = <tracking_database_port> # change to the correct port - geo_secondary['db_fdw'] = true # enable FDW - geo_postgresql['enable'] = false # don't use internal managed instance - ``` + geo_secondary['db_host'] = '<tracking_database_host>' + geo_secondary['db_port'] = <tracking_database_port> # change to the correct port + geo_secondary['db_fdw'] = true # enable FDW + geo_postgresql['enable'] = false # don't use internal managed instance + ``` 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) 1. Run the tracking database migrations: - ```bash - gitlab-rake geo:db:create - gitlab-rake geo:db:migrate - ``` - -1. Configure the - [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) - connection and credentials: - - Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection - params to match your environment. Execute it to set up the FDW connection. - - ```bash - #!/bin/bash - - # Secondary Database connection params: - DB_HOST="<public_ip_or_vpc_private_ip>" - DB_NAME="gitlabhq_production" - DB_USER="gitlab" - DB_PASS="<your_password_here>" - DB_PORT="5432" - - # Tracking Database connection params: - GEO_DB_HOST="<public_ip_or_vpc_private_ip>" - GEO_DB_NAME="gitlabhq_geo_production" - GEO_DB_USER="gitlab_geo" - GEO_DB_PORT="5432" - - query_exec () { - gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" - } - - query_exec "CREATE EXTENSION postgres_fdw;" - query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" - query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" - query_exec "CREATE SCHEMA gitlab_secondary;" - query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" - ``` - - NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, - but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using - `psql` for AWS RDS. + ```bash + gitlab-rake geo:db:create + gitlab-rake geo:db:migrate + ``` + +1. Configure the [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + connection and credentials: + + Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection + params to match your environment. Execute it to set up the FDW connection. + + ```bash + #!/bin/bash + + # Secondary Database connection params: + DB_HOST="<public_ip_or_vpc_private_ip>" + DB_NAME="gitlabhq_production" + DB_USER="gitlab" + DB_PASS="<your_password_here>" + DB_PORT="5432" + + # Tracking Database connection params: + GEO_DB_HOST="<public_ip_or_vpc_private_ip>" + GEO_DB_NAME="gitlabhq_geo_production" + GEO_DB_USER="gitlab_geo" + GEO_DB_PORT="5432" + + query_exec () { + gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" + } + + query_exec "CREATE EXTENSION postgres_fdw;" + query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" + query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" + query_exec "CREATE SCHEMA gitlab_secondary;" + query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" + ``` + + NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, + but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using + `psql` for AWS RDS. 1. Save the file and [restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) 1. Populate the FDW tables: - ```bash - gitlab-rake geo:db:refresh_foreign_tables - ``` + ```bash + gitlab-rake geo:db:refresh_foreign_tables + ``` diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index dd1af0dbf9c..b3580a706c3 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -1,4 +1,4 @@ -# Geo Frequently Asked Questions **[PREMIUM ONLY]** +# Geo Frequently Asked Questions **(PREMIUM ONLY)** ## What are the minimum requirements to run Geo? @@ -6,8 +6,8 @@ The requirements are listed [on the index page](index.md#requirements-for-runnin ## How does Geo know which projects to sync? -On each **secondary** node, there is a read-only replicated copy of the GitLab database. -A **secondary** node also has a tracking database where it stores which projects have been synced. +On each **secondary** node, there is a read-only replicated copy of the GitLab database. +A **secondary** node also has a tracking database where it stores which projects have been synced. Geo compares the two databases to find projects that are not yet tracked. At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. @@ -15,19 +15,19 @@ At the start, this tracking database is empty, so Geo will start trying to updat For each project to sync: 1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** node. -If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. + If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. 1. The **secondary** node will update the tracking database to store the fact that it has synced projects A, B, C, etc. 1. Repeat until all projects are synced. -When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. +When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. The **secondary** node sees this event, marks the project in question as dirty, and schedules the project to be resynced. To ensure that problems with pipelines (for example, syncs failing too many times or jobs being lost) don't permanently stop projects syncing, Geo also periodically checks the tracking database for projects that are marked as dirty. This check happens when -the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. +the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. -Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. -If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. -So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. +Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. +If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. +So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. ## Can I use Geo in a disaster recovery situation? diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index 921a3ef1c7a..c737fa37077 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -1,4 +1,4 @@ -# Geo High Availability **[PREMIUM ONLY]** +# Geo High Availability **(PREMIUM ONLY)** This document describes a minimal reference architecture for running Geo in a high availability configuration. If your HA setup differs from the one @@ -50,17 +50,17 @@ The following steps enable a GitLab cluster to serve as the **primary** node. 1. Edit `/etc/gitlab/gitlab.rb` and add the following: - ```ruby - ## - ## Enable the Geo primary role - ## - roles ['geo_primary_role'] + ```ruby + ## + ## Enable the Geo primary role + ## + roles ['geo_primary_role'] - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - ``` + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -79,9 +79,9 @@ The **primary** database will require modification later, as part of A **secondary** cluster is similar to any other GitLab HA cluster, with two major differences: -* The main PostgreSQL database is a read-only replica of the **primary** node's +- The main PostgreSQL database is a read-only replica of the **primary** node's PostgreSQL database. -* There is also a single PostgreSQL database for the **secondary** cluster, +- There is also a single PostgreSQL database for the **secondary** cluster, called the "tracking database", which tracks the synchronization state of various resources. @@ -93,9 +93,9 @@ from the normal HA setup. Configure the following services, again using the non-Geo high availability documentation: -* [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high +- [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high availability. -* [NFS](../../high_availability/nfs.md) which will store data that is +- [NFS](../../high_availability/nfs.md) which will store data that is synchronized from the **primary** node. ### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node @@ -107,36 +107,36 @@ Configure the [**secondary** database](database.md) as a read-only replica of the **primary** database. Use the following as a guide. 1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the - following: - - ```ruby - ## - ## Configure the PostgreSQL role - ## - roles ['postgres_role'] - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## When running the Geo tracking database on a separate machine, disable it - ## here and allow connections from the tracking database host. And ensure - ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. - ## - geo_postgresql['enable'] = false - ``` + following: + + ```ruby + ## + ## Configure the PostgreSQL role + ## + roles ['postgres_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## When running the Geo tracking database on a separate machine, disable it + ## here and allow connections from the tracking database host. And ensure + ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. + ## + geo_postgresql['enable'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -151,47 +151,47 @@ only a single machine, rather than as a PostgreSQL cluster. Configure the tracking database. 1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the - following: - - ```ruby - ## - ## Enable the Geo secondary tracking database - ## - geo_postgresql['enable'] = true - geo_postgresql['listen_address'] = '<ip_address_of_this_host>' - geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' - - ## - ## Configure FDW connection to the replica database - ## - geo_secondary['db_fdw'] = true - geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' - geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] - gitlab_rails['db_host'] = '<replica_database_ip>' - - # Prevent reconfigure from attempting to run migrations on the replica DB - gitlab_rails['auto_migrate'] = false - - ## - ## Disable all other services that aren't needed, since we don't have a role - ## that does this. - ## - alertmanager['enable'] = false - consul['enable'] = false - gitaly['enable'] = false - gitlab_monitor['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - repmgr['enable'] = false - sidekiq['enable'] = false - unicorn['enable'] = false - ``` + following: + + ```ruby + ## + ## Enable the Geo secondary tracking database + ## + geo_postgresql['enable'] = true + geo_postgresql['listen_address'] = '<ip_address_of_this_host>' + geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' + + ## + ## Configure FDW connection to the replica database + ## + geo_secondary['db_fdw'] = true + geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' + geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] + gitlab_rails['db_host'] = '<replica_database_ip>' + + # Prevent reconfigure from attempting to run migrations on the replica DB + gitlab_rails['auto_migrate'] = false + + ## + ## Disable all other services that aren't needed, since we don't have a role + ## that does this. + ## + alertmanager['enable'] = false + consul['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -211,50 +211,50 @@ following modifications: 1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** cluster, and add the following: - ```ruby - ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role', 'application_role'] - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - - ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. - ## - geo_secondary['db_host'] = '<geo_tracking_db_host>' - geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false - - ## - ## Configure connection to the streaming replica database, if you haven't - ## already - ## - gitlab_rails['db_host'] = '<replica_database_host>' - gitlab_rails['db_password'] = '<replica_database_password>' - - ## - ## Configure connection to Redis, if you haven't already - ## - gitlab_rails['redis_host'] = '<redis_host>' - gitlab_rails['redis_password'] = '<redis_password>' - - ## - ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a - ## cluster to avoid permissions issues - ## - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role', 'application_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '<geo_tracking_db_host>' + geo_secondary['db_password'] = '<geo_tracking_db_password>' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '<replica_database_host>' + gitlab_rails['db_password'] = '<replica_database_password>' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '<redis_host>' + gitlab_rails['redis_password'] = '<redis_password>' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` NOTE: **Note:** If you had set up PostgreSQL cluster using the omnibus package and you had set @@ -270,15 +270,15 @@ After making these changes [Reconfigure GitLab][gitlab-reconfigure] so the chang On the secondary the following GitLab frontend services will be enabled: -* geo-logcursor -* gitlab-pages -* gitlab-workhorse -* logrotate -* nginx -* registry -* remote-syslog -* sidekiq -* unicorn +- geo-logcursor +- gitlab-pages +- gitlab-workhorse +- logrotate +- nginx +- registry +- remote-syslog +- sidekiq +- unicorn Verify these services by running `sudo gitlab-ctl status` on the frontend application servers. diff --git a/doc/administration/geo/replication/img/geo_architecture.png b/doc/administration/geo/replication/img/geo_architecture.png Binary files differindex d318cd5d0f4..aac63be41ff 100644 --- a/doc/administration/geo/replication/img/geo_architecture.png +++ b/doc/administration/geo/replication/img/geo_architecture.png diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 54377f7ae4e..dbd466b906d 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -1,6 +1,12 @@ -# Geo Replication **[PREMIUM ONLY]** +# Replication (Geo) **(PREMIUM ONLY)** -Geo is the solution for widely distributed development teams. +> - Introduced in GitLab Enterprise Edition 8.9. +> - Using Geo in combination with +> [High Availability](../../high_availability/README.md) +> is considered **Generally Available** (GA) in +> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. + +Replication with Geo is the solution for widely distributed development teams. ## Overview @@ -8,14 +14,8 @@ Fetching large repositories can take a long time for teams located far from a si Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development. -> - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed). -> - Introduced in GitLab Enterprise Edition 8.9. -> - We recommend you use: -> - At least GitLab Enterprise Edition 10.0 for basic Geo features. -> - The latest version for a better experience. -> - Make sure that all nodes run the same GitLab version. -> - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md). -> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. +NOTE: **Note:** +Check the [requirements](#requirements-for-running-geo) carefully before setting up Geo. For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). @@ -80,8 +80,8 @@ In this diagram: - If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios. - A **secondary** node performs different type of synchronizations against the **primary** node, using a special authorization protected by JWT: - - Repositories are cloned/updated via Git over HTTPS. - - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. + - Repositories are cloned/updated via Git over HTTPS. + - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. From the perspective of a user performing Git operations: @@ -107,10 +107,17 @@ The following are required to run Geo: - An operating system that supports OpenSSH 6.9+ (needed for [fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md)) The following operating systems are known to ship with a current version of OpenSSH: - - [CentOS](https://www.centos.org) 7.4+ - - [Ubuntu](https://www.ubuntu.com) 16.04+ + - [CentOS](https://www.centos.org) 7.4+ + - [Ubuntu](https://www.ubuntu.com) 16.04+ - PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ +- All nodes must run the same GitLab version. + +Additionally, check GitLab's [minimum requirements](../../../install/requirements.md), +and we recommend you use: + +- At least GitLab Enterprise Edition 10.0 for basic Geo features. +- The latest version for a better experience. ### Firewall rules @@ -178,7 +185,7 @@ The steps below should be followed in the order they appear. **Make sure the Git If you installed GitLab using the Omnibus packages (highly recommended): -1. [Install GitLab Enterprise Edition](https://about.gitlab.com/installation/) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. 1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). 1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** nodes. @@ -239,35 +246,48 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. -### Limitations on replication - -Only the following items are replicated to the **secondary** node: - -- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata. -- Project repositories. -- Project wiki repositories. -- User uploads. For example, attachments to issues, merge requests, epics, and avatars. -- CI job artifacts and traces. +### Limitations on replication/verification + +The following table lists the GitLab features along with their replication +and verification status on a **secondary** node. + +You can keep track of the progress to include the missing items in: + +- [ee-893](https://gitlab.com/groups/gitlab-org/-/epics/893). +- [ee-1430](https://gitlab.com/groups/gitlab-org/-/epics/1430). + +| Feature | Replicated | Verified | +|-----------|------------|----------| +| All database content (e.g. snippets, epics, issues, merge requests, groups, and project metadata) | Yes | Yes | +| Project repository | Yes | Yes | +| Project wiki repository | Yes | Yes | +| Project designs repository | No | No | +| Uploads (e.g. attachments to issues, merge requests, epics, and avatars) | Yes | Yes, only on transfer, or manually (1) | +| LFS Objects | Yes | Yes, only on transfer, or manually (1) | +| CI job artifacts (other than traces) | Yes | No, only manually (1) | +| Archived traces | Yes | Yes, only on transfer, or manually (1) | +| Personal snippets | Yes | Yes | +| Version-controlled personal snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No | +| Project snippets | Yes | Yes | +| Version-controlled project snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No | +| Object pools for forked project deduplication | No | No | +| [Server-side Git Hooks](../../custom_hooks.md) | No | No | +| [Elasticsearch integration](../../../integration/elasticsearch.md) | No | No | +| [GitLab Pages](../../pages/index.md) | No | No | +| [Container Registry](../../container_registry.md) ([track progress](https://gitlab.com/gitlab-org/gitlab-ee/issues/2870)) | No | No | +| [NPM Registry](../../npm_registry.md) | No | No | +| [Maven Packages](../../maven_packages.md) | No | No | +| [External merge request diffs](../../merge_request_diffs.md) | No, if they are on-disk | No | +| Content in object storage ([track progress](https://gitlab.com/groups/gitlab-org/-/epics/1526)) | No | No | + +1. The integrity can be verified manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. DANGER: **DANGER** -Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**. - -### Examples of data not replicated - -Take special note that these examples of GitLab features are both: - -- Commonly used. -- **Not** replicated by Geo at present. - -Examples include: - -- [Elasticsearch integration](../../../integration/elasticsearch.md). -- [Container Registry](../../container_registry.md). [Object Storage](object_storage.md) can mitigate this. -- [GitLab Pages](../../pages/index.md). -- [Mattermost integration](https://docs.gitlab.com/omnibus/gitlab-mattermost/). - -CAUTION: **Caution:** -If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means. +Features not on this list, or with **No** in the **Replicated** column, +are not replicated on the **secondary** node. Failing over without manually +replicating data from those features will cause the data to be **lost**. +If you wish to use those features on a **secondary** node, or to execute a failover +successfully, you must replicate their data using some other means. ## Frequently Asked Questions diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index c3c11dbaf1e..878b67a8f8e 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -1,4 +1,4 @@ -# Geo with Object storage **[PREMIUM ONLY]** +# Geo with Object storage **(PREMIUM ONLY)** Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). @@ -34,10 +34,10 @@ the bucket used by **secondary** nodes. If you are using Google Cloud Storage, consider using [Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional). -Or you can use the [Storage Transfer Service](https://cloud.google.com/storage/transfer/), +Or you can use the [Storage Transfer Service](https://cloud.google.com/storage-transfer/docs/), although this only supports daily synchronization. For manual synchronization, or scheduled by `cron`, please have a look at: -- [`s3cmd sync`](http://s3tools.org/s3cmd-sync) +- [`s3cmd sync`](https://s3tools.org/s3cmd-sync) - [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync) diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index b190fe7d42d..e24eb2bd428 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,4 +1,4 @@ -# Removing secondary Geo nodes **[PREMIUM ONLY]** +# Removing secondary Geo nodes **(PREMIUM ONLY)** **Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: @@ -10,41 +10,42 @@ Once removed from the Geo admin page, you must stop and uninstall the **secondar 1. On the **secondary** node, stop GitLab: - ```bash - sudo gitlab-ctl stop - ``` + ```bash + sudo gitlab-ctl stop + ``` + 1. On the **secondary** node, uninstall GitLab: - ```bash - # Stop gitlab and remove its supervision process - sudo gitlab-ctl uninstall - - # Debian/Ubuntu - sudo dpkg --remove gitlab-ee - - # Redhat/Centos - sudo rpm --erase gitlab-ee - ``` + ```bash + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall + + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee + + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: 1. On the **primary** node, start a PostgreSQL console session: - ```bash - sudo gitlab-psql - ``` - - NOTE: **Note:** - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + ```bash + sudo gitlab-psql + ``` + + NOTE: **Note:** + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. - ```sql - SELECT * FROM pg_replication_slots; - ``` - + ```sql + SELECT * FROM pg_replication_slots; + ``` + 1. Remove the replication slot for the **secondary** node: - ```sql - SELECT pg_drop_replication_slot('<name_of_slot>'); - ``` + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index cd54e2dc8c4..ed3f1faa93e 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -1,4 +1,4 @@ -# Geo security review (Q&A) **[PREMIUM ONLY]** +# Geo security review (Q&A) **(PREMIUM ONLY)** The following security review of the Geo feature set focuses on security aspects of the feature as they apply to customers running their own GitLab @@ -115,7 +115,7 @@ questions from [owasp.org](https://www.owasp.org). ### What operating systems support the application? - Geo imposes no additional restrictions on operating system (see the - [GitLab installation](https://about.gitlab.com/installation/) page for more + [GitLab installation](https://about.gitlab.com/install/) page for more details), however we recommend using the operating systems listed in the [Geo documentation](index.md#requirements-for-running-geo). ### What details regarding required OS components and lock‐down needs have been defined? diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index c5bdd36ba70..fe1557fd8b5 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1,15 +1,23 @@ -# Geo Troubleshooting **[PREMIUM ONLY]** - -NOTE: **Note:** -This list is an attempt to document all the moving parts that can go wrong. -We are working into getting all this steps verified automatically in a -rake task in the future. +# Geo Troubleshooting **(PREMIUM ONLY)** Setting up Geo requires careful attention to details and sometimes it's easy to -miss a step. Here is a list of questions you should ask to try to detect -what you need to fix (all commands and path locations are for Omnibus installs): +miss a step. + +Here is a list of steps you should take to attempt to fix problem: + +- Perform [basic troubleshooting](#basic-troubleshooting). +- Fix any [replication errors](#fixing-replication-errors). +- Fix any [Foreign Data Wrapper](#fixing-foreign-data-wrapper-errors) errors. +- Fix any [common](#fixing-common-errors) errors. + +## Basic troubleshooting + +Before attempting more advanced troubleshooting: -## First check the health of the **secondary** node +- Check [the health of the **secondary** node](#check-the-health-of-the-secondary-node). +- Check [if PostgreSQL replication is working](#check-if-postgresql-replication-is-working). + +### Check the health of the **secondary** node Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in your browser. We perform the following health checks on each **secondary** node @@ -23,10 +31,12 @@ to help identify if something is wrong:  -For information on how to resolve common errors reported from the UI, see [common errors](#common-errors). +For information on how to resolve common errors reported from the UI, see +[Fixing Common Errors](#fixing-common-errors). If the UI is not working, or you are unable to log in, you can run the Geo health check manually to get this information as well as a few more details. + This rake task can be run on an app node in the **primary** or **secondary** Geo nodes: @@ -36,7 +46,7 @@ sudo gitlab-rake gitlab:geo:check Example output: -``` +```text Checking Geo ... GitLab Geo is available ... yes @@ -68,7 +78,7 @@ sudo gitlab-rake geo:status Example output: -``` +```text http://secondary.example.com/ ----------------------------------------------------- GitLab Version: 11.10.4-ee @@ -89,16 +99,21 @@ http://secondary.example.com/ Last status report was: 2 minutes ago ``` -## Is Postgres replication working? +### Check if PostgreSQL replication is working + +To check if PostgreSQL replication is working, check if: -### Are my nodes pointing to the correct database instance? +- [Nodes are pointing to the correct database instance](#are-nodes-pointing-to-the-correct-database-instance). +- [Geo can detect the current node correctly](#can-geo-detect-the-current-node-correctly). + +#### Are nodes pointing to the correct database instance? You should make sure your **primary** Geo node points to the instance with writing permissions. Any **secondary** nodes should point only to read-only instances. -### Can Geo detect my current node correctly? +#### Can Geo detect the current node correctly? Geo uses the defined node from the **Admin Area > Geo** screen, and tries to match it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file. @@ -112,29 +127,38 @@ sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect" and expect something like: -``` +```ruby #<GeoNode id: 2, schema: "https", host: "gitlab.example.com", port: 443, relative_url_root: "", primary: false, ...> ``` By running the command above, `primary` should be `true` when executed in the **primary** node, and `false` on any **secondary** node. -## How do I fix the message, "ERROR: replication slots can only be used if max_replication_slots > 0"? +## Fixing replication errors + +The following sections outline troubleshooting steps for fixing replication +errors. + +### Message: "ERROR: replication slots can only be used if max_replication_slots > 0"? This means that the `max_replication_slots` PostgreSQL variable needs to be set on the **primary** database. In GitLab 9.4, we have made this setting default to 1. You may need to increase this value if you have more -**secondary** nodes. Be sure to restart PostgreSQL for this to take +**secondary** nodes. + +Be sure to restart PostgreSQL for this to take effect. See the [PostgreSQL replication setup][database-pg-replication] guide for more details. -## How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? +### Message: "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? This occurs when PostgreSQL does not have a replication slot for the -**secondary** node by that name. You may want to rerun the [replication +**secondary** node by that name. + +You may want to rerun the [replication process](database.md) on the **secondary** node . -## How do I fix the message, "Command exceeded allowed execution time" when setting up replication? +### Message: "Command exceeded allowed execution time" when setting up replication? This may happen while [initiating the replication process][database-start-replication] on the **secondary** node, and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes). @@ -153,24 +177,24 @@ sudo gitlab-ctl \ This will give the initial replication up to six hours to complete, rather than the default thirty minutes. Adjust as required for your installation. -## How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" +### Message: "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" Determine if you have any unused replication slots in the **primary** database. This can cause large amounts of log data to build up in `pg_xlog`. Removing the unused slots can reduce the amount of space used in the `pg_xlog`. 1. Start a PostgreSQL console session: - ```sh - sudo gitlab-psql gitlabhq_production - ``` + ```sh + sudo gitlab-psql gitlabhq_production + ``` - > Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + Note: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. View your replication slots with: - ```sql - SELECT * FROM pg_replication_slots; - ``` + ```sql + SELECT * FROM pg_replication_slots; + ``` Slots where `active` is `f` are not active. @@ -180,15 +204,16 @@ Slots where `active` is `f` are not active. - If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the PostgreSQL console session: - ```sql - SELECT pg_drop_replication_slot('<name_of_extra_slot>'); - ``` + ```sql + SELECT pg_drop_replication_slot('<name_of_extra_slot>'); + ``` -## Very large repositories never successfully synchronize on the **secondary** node +### Very large repositories never successfully synchronize on the **secondary** node GitLab places a timeout on all repository clones, including project imports and Geo synchronization operations. If a fresh `git clone` of a repository on the primary takes more than a few minutes, you may be affected by this. + To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb` on the **secondary** node: @@ -205,96 +230,100 @@ sudo gitlab-ctl reconfigure This will increase the timeout to three hours (10800 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. -## How to reset Geo **secondary** node replication +### Resetting Geo **secondary** node replication If you get a **secondary** node in a broken state and want to reset the replication state, to start again from scratch, there are a few steps that can help you: 1. Stop Sidekiq and the Geo LogCursor - It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and - wait until the current jobs to finish processing. + It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and + wait until the current jobs to finish processing. - You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** - when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. + You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** + when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. - ```sh - gitlab-ctl status sidekiq - # run: sidekiq: (pid 10180) <- this is the PID you will use - kill -TSTP 10180 # change to the correct PID + ```sh + gitlab-ctl status sidekiq + # run: sidekiq: (pid 10180) <- this is the PID you will use + kill -TSTP 10180 # change to the correct PID - gitlab-ctl stop sidekiq - gitlab-ctl stop geo-logcursor - ``` + gitlab-ctl stop sidekiq + gitlab-ctl stop geo-logcursor + ``` - You can watch sidekiq logs to know when sidekiq jobs processing have finished: + You can watch sidekiq logs to know when sidekiq jobs processing have finished: - ```sh - gitlab-ctl tail sidekiq - ``` + ```sh + gitlab-ctl tail sidekiq + ``` 1. Rename repository storage folders and create new ones - ```sh - mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old - mkdir -p /var/opt/gitlab/git-data/repositories - chown git:git /var/opt/gitlab/git-data/repositories - ``` + ```sh + mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old + mkdir -p /var/opt/gitlab/git-data/repositories + chown git:git /var/opt/gitlab/git-data/repositories + ``` - TIP: **Tip** - You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future - as soon as you confirmed that you don't need it anymore, to save disk space. + TIP: **Tip** + You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future + as soon as you confirmed that you don't need it anymore, to save disk space. 1. _(Optional)_ Rename other data folders and create new ones - CAUTION: **Caution**: - You may still have files on the **secondary** node that have been removed from **primary** node but - removal have not been reflected. If you skip this step, they will never be removed - from this Geo node. + CAUTION: **Caution**: + You may still have files on the **secondary** node that have been removed from **primary** node but + removal have not been reflected. If you skip this step, they will never be removed + from this Geo node. - Any uploaded content like file attachments, avatars or LFS objects are stored in a - subfolder in one of the two paths below: + Any uploaded content like file attachments, avatars or LFS objects are stored in a + subfolder in one of the two paths below: - 1. /var/opt/gitlab/gitlab-rails/shared - 1. /var/opt/gitlab/gitlab-rails/uploads + - /var/opt/gitlab/gitlab-rails/shared + - /var/opt/gitlab/gitlab-rails/uploads - To rename all of them: + To rename all of them: - ```sh - gitlab-ctl stop + ```sh + gitlab-ctl stop - mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old - mkdir -p /var/opt/gitlab/gitlab-rails/shared + mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old + mkdir -p /var/opt/gitlab/gitlab-rails/shared - mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old - mkdir -p /var/opt/gitlab/gitlab-rails/uploads - ``` + mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old + mkdir -p /var/opt/gitlab/gitlab-rails/uploads + ``` - Reconfigure in order to recreate the folders and make sure permissions and ownership - are correctly + Reconfigure in order to recreate the folders and make sure permissions and ownership + are correctly - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Reset the Tracking Database - ```sh - gitlab-rake geo:db:reset - ``` + ```sh + gitlab-rake geo:db:reset + ``` 1. Restart previously stopped services - ```sh - gitlab-ctl start - ``` + ```sh + gitlab-ctl start + ``` + +## Fixing Foreign Data Wrapper errors -## How do I fix a "Foreign Data Wrapper (FDW) is not configured" error? +This section documents ways to fix potential Foreign Data Wrapper errors. + +### "Foreign Data Wrapper (FDW) is not configured" error When setting up Geo, you might see this warning in the `gitlab-rake gitlab:geo:check` output: -``` +```text GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured ``` @@ -302,12 +331,12 @@ There are a few key points to remember: 1. The FDW settings are configured on the Geo **tracking** database. 1. The configured foreign server enables a login to the Geo -**secondary**, read-only database. + **secondary**, read-only database. By default, the Geo secondary and tracking database are running on the same host on different ports. That is, 5432 and 5431 respectively. -### Checking configuration +#### Checking configuration NOTE: **Note:** The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5. @@ -316,110 +345,108 @@ To check the configuration: 1. Enter the database console: - ```sh - gitlab-geo-psql - ``` + ```sh + gitlab-geo-psql + ``` 1. Check whether any tables are present. If everything is working, you -should see something like this: - - ```sql - gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; - foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_n - ame - -------------------------+----------------------+-------------------------------------------------+-------------------------+----------------- - ---- - gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary - <snip> - ``` - - However, if the query returns with `0 rows`, then continue onto the next steps. + should see something like this: + + ```sql + gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; + foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name + -------------------------+----------------------+-------------------------------------------------+-------------------------+--------------------- + gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary + <snip> + ``` + + However, if the query returns with `0 rows`, then continue onto the next steps. 1. Check that the foreign server mapping is correct via `\des+`. The results should look something like this: - ```sql - gitlabhq_geo_production=# \des+ - List of foreign servers - -[ RECORD 1 ]--------+------------------------------------------------------------ - Name | gitlab_secondary - Owner | gitlab-psql - Foreign-data wrapper | postgres_fdw - Access privileges | "gitlab-psql"=U/"gitlab-psql" + - | gitlab_geo=U/"gitlab-psql" - Type | - Version | - FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') - Description | - ``` - - NOTE: **Note:** Pay particular attention to the host and port under - FDW options. That configuration should point to the Geo secondary - database. - - If you need to experiment with changing the host or password, the - following queries demonstrate how: - - ```sql - ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); - ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); - ``` - - If you change the host and/or port, you will also have to adjust the - following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl - reconfigure`: - - - `gitlab_rails['db_host']` - - `gitlab_rails['db_port']` + ```sql + gitlabhq_geo_production=# \des+ + List of foreign servers + -[ RECORD 1 ]--------+------------------------------------------------------------ + Name | gitlab_secondary + Owner | gitlab-psql + Foreign-data wrapper | postgres_fdw + Access privileges | "gitlab-psql"=U/"gitlab-psql" + + | gitlab_geo=U/"gitlab-psql" + Type | + Version | + FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') + Description | + ``` + + NOTE: **Note:** Pay particular attention to the host and port under + FDW options. That configuration should point to the Geo secondary + database. + + If you need to experiment with changing the host or password, the + following queries demonstrate how: + + ```sql + ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); + ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); + ``` + + If you change the host and/or port, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_host']` + - `gitlab_rails['db_port']` 1. Check that the user mapping is configured properly via `\deu+`: - ```sql - gitlabhq_geo_production=# \deu+ - List of user mappings - Server | User name | FDW Options - ------------------+------------+-------------------------------------------------------------------------------- - gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') - (1 row) - ``` + ```sql + gitlabhq_geo_production=# \deu+ + List of user mappings + Server | User name | FDW Options + ------------------+------------+-------------------------------------------------------------------------------- + gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') + (1 row) + ``` - Make sure the password is correct. You can test that logins work by running `psql`: + Make sure the password is correct. You can test that logins work by running `psql`: - ```sh - # Connect to the tracking database as the `gitlab_geo` user - sudo \ - -u git /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/geo-postgresql \ - -p 5431 \ - -U gitlab_geo \ - -W \ - -d gitlabhq_geo_production - ``` + ```sh + # Connect to the tracking database as the `gitlab_geo` user + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` - If you need to correct the password, the following query shows how: + If you need to correct the password, the following query shows how: - ```sql - ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); - ``` + ```sql + ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); + ``` - If you change the user or password, you will also have to adjust the - following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl - reconfigure`: + If you change the user or password, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: - - `gitlab_rails['db_username']` - - `gitlab_rails['db_password']` + - `gitlab_rails['db_username']` + - `gitlab_rails['db_password']` - If you are using [PgBouncer in front of the secondary - database](database.md#pgbouncer-support-optional), be sure to update - the following settings: + If you are using [PgBouncer in front of the secondary + database](database.md#pgbouncer-support-optional), be sure to update + the following settings: - - `geo_postgresql['fdw_external_user']` - - `geo_postgresql['fdw_external_password']` + - `geo_postgresql['fdw_external_user']` + - `geo_postgresql['fdw_external_password']` -### Manual reload of FDW schema +#### Manual reload of FDW schema If you're still unable to get FDW working, you may want to try a manual reload of the FDW schema. To manually reload the FDW schema: @@ -427,41 +454,66 @@ reload of the FDW schema. To manually reload the FDW schema: 1. On the node running the Geo tracking database, enter the PostgreSQL console via the `gitlab_geo` user: - ```sh - sudo \ - -u git /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/geo-postgresql \ - -p 5431 \ - -U gitlab_geo \ - -W \ - -d gitlabhq_geo_production - ``` + ```sh + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` - Be sure to adjust the port and hostname for your configuration. You - may be asked to enter a password. + Be sure to adjust the port and hostname for your configuration. You + may be asked to enter a password. 1. Reload the schema via: - ```sql - DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; - CREATE SCHEMA gitlab_secondary; - GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; - IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; - ``` + ```sql + DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; + CREATE SCHEMA gitlab_secondary; + GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; + IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; + ``` 1. Test that queries work: - ```sql - SELECT * from information_schema.foreign_tables; - SELECT * FROM gitlab_secondary.projects limit 1; - ``` + ```sql + SELECT * from information_schema.foreign_tables; + SELECT * FROM gitlab_secondary.projects limit 1; + ``` [database-start-replication]: database.md#step-3-initiate-the-replication-process [database-pg-replication]: database.md#postgresql-replication -## Common errors +### "Geo database has an outdated FDW remote schema" error + +GitLab can error with a `Geo database has an outdated FDW remote schema` message. + +For example: + +```text +Geo database has an outdated FDW remote schema. It contains 229 of 236 expected tables. Please refer to Geo Troubleshooting. +``` + +To resolve this, run the following command: + +```sh +sudo gitlab-rake geo:db:refresh_foreign_tables +``` + +## Expired artifacts -This section documents common errors reported in the admin UI and how to fix them. +If you notice for some reason there are more artifacts on the Geo +secondary node than on the Geo primary node, you can use the rake task +to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). + +On a Geo **secondary** node, this command will also clean up all Geo +registry record related to the orphan files on disk. + +## Fixing common errors + +This section documents common errors reported in the Admin UI and how to fix them. ### Geo database configuration file is missing @@ -470,9 +522,22 @@ GitLab cannot find or doesn't have permission to access the `database_geo.yml` c In an Omnibus GitLab installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. If it doesn't exist or inadvertent changes have been made to it, run `sudo gitlab-ctl reconfigure` to restore it to its correct state. - If this path is mounted on a remote volume, please check your volume configuration and that it has correct permissions. +### An existing tracking database cannot be reused + +Geo cannot reuse an existing tracking database. + +It is safest to use a fresh secondary, or reset the whole secondary by following +[Resetting Geo secondary node replication](#resetting-geo-secondary-node-replication). + +If you are not concerned about possible orphaned directories and files, then you +can simply reset the existing tracking database with: + +```sh +sudo gitlab-rake geo:db:reset +``` + ### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node. This error refers to a problem with the database replica on a **secondary** node, @@ -503,7 +568,7 @@ Make sure you follow the [Geo database replication](database.md) instructions fo If you are using GitLab Omnibus installation, something might have failed during upgrade. You can: -- Run `sudo gitlab-ctl reconfigure`. +- Run `sudo gitlab-ctl reconfigure`. - Manually trigger the database migration by running: `sudo gitlab-rake geo:db:migrate` as root on the **secondary** node. ### Geo database is not configured to use Foreign Data Wrapper @@ -511,4 +576,4 @@ If you are using GitLab Omnibus installation, something might have failed during This error means the Geo Tracking Database doesn't have the FDW server and credentials configured. -See [How do I fix a "Foreign Data Wrapper (FDW) is not configured" error?](#how-do-i-fix-a-foreign-data-wrapper-fdw-is-not-configured-error). +See ["Foreign Data Wrapper (FDW) is not configured" error?](#foreign-data-wrapper-fdw-is-not-configured-error). diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 1943f2230df..3ee9937774a 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -1,4 +1,4 @@ -# Tuning Geo **[PREMIUM ONLY]** +# Tuning Geo **(PREMIUM ONLY)** ## Changing the sync capacity values diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 933a75c47d8..39174780e24 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -1,4 +1,4 @@ -# Updating the Geo nodes **[PREMIUM ONLY]** +# Updating the Geo nodes **(PREMIUM ONLY)** Depending on which version of Geo you are updating to/from, there may be different steps. @@ -10,10 +10,33 @@ all you need to do is update GitLab itself: 1. Log into each node (**primary** and **secondary** nodes). 1. [Update GitLab][update]. -1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when - the tracking database is enabled. 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. +### Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo raketask on all nodes, everything should be green: + + ```sh + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** node's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** node and see if it + is received by **secondary** nodes. + +## Upgrading to GitLab 12.1 + +By default, GitLab 12.1 will attempt to automatically upgrade the embedded PostgreSQL server to 10.7 from 9.6. Please see [the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) for the recommended procedure. + +This can be temporarily disabled by running the following before ugprading: + +```sh +sudo touch /etc/gitlab/disable-postgresql-upgrade +``` + ## Upgrading to GitLab 10.8 Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node: @@ -30,70 +53,70 @@ We now require this change as we use this password to enable the Foreign Data Wr the Geo Tracking Database. We are also improving security by disabling the use of **trust** authentication method. -1. **[primary]** Login to your **primary** node and run: +1. **(primary)** Login to your **primary** node and run: - ```sh - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` - Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: + Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this - ``` + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this + ``` -1. **[primary]** Reconfigure and restart: +1. **(primary)** Reconfigure and restart: - ```sh - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` -1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: +1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' - # Enable Foreign Data Wrapper - geo_secondary['db_fdw'] = true + # Enable Foreign Data Wrapper + geo_secondary['db_fdw'] = true - # Secondary address in CIDR format, for example '5.6.7.8/32' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - ``` + # Secondary address in CIDR format, for example '5.6.7.8/32' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + ``` - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this - ``` + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this + ``` -1. **[secondary]** Reconfigure and restart: +1. **(secondary)** Reconfigure and restart: - ```sh - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` ## Upgrading to GitLab 10.5 @@ -169,11 +192,11 @@ After you've verified that HTTP/HTTPS replication is working, you should remove the now-unused SSH keys from your secondaries, as they may cause problems if the **secondary** node if ever promoted to a **primary** node: -1. **[secondary]** Login to **all** your **secondary** nodes and run: +1. **(secondary)** Login to **all** your **secondary** nodes and run: - ```ruby - sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub - ``` + ```ruby + sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub + ``` ### Hashed Storage @@ -236,12 +259,12 @@ instructions below. When in doubt, it does not hurt to do a resync. The easiest way to do this in Omnibus is the following: - 1. Make sure you have Omnibus GitLab on the **primary** server. - 1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. - 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. - 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. - 1. Install GitLab on the **secondary** server. - 1. Re-run the [database replication process][database-replication]. +1. Make sure you have Omnibus GitLab on the **primary** server. +1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. +1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. +1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. +1. Install GitLab on the **secondary** server. +1. Re-run the [database replication process](database.md#step-3-initiate-the-replication-process). ## Special update notes for 9.0.x @@ -260,174 +283,156 @@ Make sure to follow the steps in the exact order as they appear below and pay extra attention in what node (either **primary** or **secondary**) you execute them! Each step is prepended with the relevant node for better clarity: -1. **[secondary]** Login to **all** your **secondary** nodes and stop all services: +1. **(secondary)** Login to **all** your **secondary** nodes and stop all services: - ```ruby - sudo gitlab-ctl stop - ``` + ```ruby + sudo gitlab-ctl stop + ``` -1. **[secondary]** Make a backup of the `recovery.conf` file on **all** +1. **(secondary)** Make a backup of the `recovery.conf` file on **all** **secondary** nodes to preserve PostgreSQL's credentials: - ```sh - sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ - ``` + ```sh + sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ + ``` -1. **[primary]** Update the **primary** node to GitLab 9.0 following the +1. **(primary)** Update the **primary** node to GitLab 9.0 following the [regular update docs][update]. At the end of the update, the **primary** node will be running with PostgreSQL 9.6. -1. **[primary]** To prevent a de-synchronization of the repository replication, +1. **(primary)** To prevent a de-synchronization of the repository replication, stop all services except `postgresql` as we will use it to re-initialize the **secondary** node's database: - ```sh - sudo gitlab-ctl stop - sudo gitlab-ctl start postgresql - ``` + ```sh + sudo gitlab-ctl stop + sudo gitlab-ctl start postgresql + ``` -1. **[secondary]** Run the following steps on each of the **secondary** nodes: +1. **(secondary)** Run the following steps on each of the **secondary** nodes: - 1. **[secondary]** Stop all services: + 1. **(secondary)** Stop all services: - ```sh - sudo gitlab-ctl stop - ``` + ```sh + sudo gitlab-ctl stop + ``` - 1. **[secondary]** Prevent running database migrations: + 1. **(secondary)** Prevent running database migrations: - ```sh - sudo touch /etc/gitlab/skip-auto-migrations - ``` + ```sh + sudo touch /etc/gitlab/skip-auto-migrations + ``` - 1. **[secondary]** Move the old database to another directory: + 1. **(secondary)** Move the old database to another directory: - ```sh - sudo mv /var/opt/gitlab/postgresql{,.bak} - ``` + ```sh + sudo mv /var/opt/gitlab/postgresql{,.bak} + ``` - 1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update]. - At the end of the update, the node will be running with PostgreSQL 9.6. + 1. **(secondary)** Update to GitLab 9.0 following the [regular update docs][update]. + At the end of the update, the node will be running with PostgreSQL 9.6. - 1. **[secondary]** Make sure all services are up: + 1. **(secondary)** Make sure all services are up: - ```sh - sudo gitlab-ctl start - ``` + ```sh + sudo gitlab-ctl start + ``` - 1. **[secondary]** Reconfigure GitLab: + 1. **(secondary)** Reconfigure GitLab: - ```sh - sudo gitlab-ctl reconfigure - ``` + ```sh + sudo gitlab-ctl reconfigure + ``` - 1. **[secondary]** Run the PostgreSQL upgrade command: + 1. **(secondary)** Run the PostgreSQL upgrade command: - ```sh - sudo gitlab-ctl pg-upgrade - ``` + ```sh + sudo gitlab-ctl pg-upgrade + ``` - 1. **[secondary]** See the stored credentials for the database that you will - need to re-initialize the replication: + 1. **(secondary)** See the stored credentials for the database that you will + need to re-initialize the replication: - ```sh - sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf - ``` + ```sh + sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf + ``` - 1. **[secondary]** Create the `replica.sh` script as described in the - [database configuration document][database-source-replication]. + 1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the + embedded paths if necessary: - 1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the - embedded paths if necessary: + ``` + #!/bin/bash - ``` - #!/bin/bash + PORT="5432" + USER="gitlab_replicator" + echo --------------------------------------------------------------- + echo WARNING: Make sure this script is run from the secondary server + echo --------------------------------------------------------------- + echo + echo Enter the IP or FQDN of the primary PostgreSQL server + read HOST + echo Enter the password for $USER@$HOST + read -s PASSWORD + echo Enter the required sslmode + read SSLMODE - PORT="5432" - USER="gitlab_replicator" - echo --------------------------------------------------------------- - echo WARNING: Make sure this script is run from the secondary server - echo --------------------------------------------------------------- - echo - echo Enter the IP or FQDN of the primary PostgreSQL server - read HOST - echo Enter the password for $USER@$HOST - read -s PASSWORD - echo Enter the required sslmode - read SSLMODE + echo Stopping PostgreSQL and all GitLab services + sudo service gitlab stop + sudo service postgresql stop - echo Stopping PostgreSQL and all GitLab services - sudo service gitlab stop - sudo service postgresql stop + echo Backing up postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ - echo Backing up postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ + echo Cleaning up old cluster directory + sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data - echo Cleaning up old cluster directory - sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data + echo Starting base backup as the replicator user + echo Enter the password for $USER@$HOST + sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P - echo Starting base backup as the replicator user - echo Enter the password for $USER@$HOST - sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P + echo Writing recovery.conf file + sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ + standby_mode = 'on' + primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' + _EOF1_ + " - echo Writing recovery.conf file - sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ - standby_mode = 'on' - primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' - _EOF1_ - " + echo Restoring postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ - echo Restoring postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ + echo Starting PostgreSQL + sudo service postgresql start + ``` - echo Starting PostgreSQL - sudo service postgresql start - ``` + 1. **(secondary)** Run the recovery script using the credentials from the + previous step: - 1. **[secondary]** Run the recovery script using the credentials from the - previous step: + ```sh + sudo bash /tmp/replica.sh + ``` - ```sh - sudo bash /tmp/replica.sh - ``` + 1. **(secondary)** Reconfigure GitLab: - 1. **[secondary]** Reconfigure GitLab: + ```sh + sudo gitlab-ctl reconfigure + ``` - ```sh - sudo gitlab-ctl reconfigure - ``` + 1. **(secondary)** Start all services: - 1. **[secondary]** Start all services: + ```sh + sudo gitlab-ctl start + ``` - ```sh - sudo gitlab-ctl start - ``` + 1. **(secondary)** Repeat the steps for the remaining **secondary** nodes. - 1. **[secondary]** Repeat the steps for the remaining **secondary** nodes. - -1. **[primary]** After all **secondary** nodes are updated, start all services in +1. **(primary)** After all **secondary** nodes are updated, start all services in **primary** node: - ```sh - sudo gitlab-ctl start - ``` - -## Check status after updating - -Now that the update process is complete, you may want to check whether -everything is working correctly: - -1. Run the Geo raketask on all nodes, everything should be green: - - ```sh - sudo gitlab-rake gitlab:geo:check - ``` - -1. Check the **primary** node's Geo dashboard for any errors. -1. Test the data replication by pushing code to the **primary** node and see if it - is received by **secondary** nodes. + ```sh + sudo gitlab-ctl start + ``` -## Update tracking database on **secondary** node +### Update tracking database on **secondary** node After updating a **secondary** node, you might need to run migrations on the tracking database. The tracking database was added in GitLab 9.1, @@ -435,9 +440,9 @@ and it is required since 10.0. 1. Run database migrations on tracking database: - ```sh - sudo gitlab-rake geo:db:migrate - ``` + ```sh + sudo gitlab-rake geo:db:migrate + ``` 1. Repeat this step for each **secondary** node. diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index f1f1fe48748..55b5d486676 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,6 +1,6 @@ [//]: # (Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) -# Using a Geo Server **[PREMIUM ONLY]** +# Using a Geo Server **(PREMIUM ONLY)** After you set up the [database replication and configure the Geo nodes][req], use your closest GitLab node as you would a normal standalone GitLab instance. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 53a85dfad6c..150494c47e5 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -2,127 +2,124 @@ [Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that provides high-level RPC access to Git repositories. Without it, no other -components can read or write Git data. +components can read or write Git data. GitLab components that access Git +repositories (gitlab-rails, gitlab-shell, gitlab-workhorse, etc.) act as clients +to Gitaly. End users do not have direct access to Gitaly. -GitLab components that access Git repositories (gitlab-rails, -gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do -not have direct access to Gitaly. +In the rest of this page, Gitaly server is referred to the standalone node that +only runs Gitaly, and Gitaly client to the GitLab Rails node that runs all other +processes except Gitaly. ## Configuring Gitaly The Gitaly service itself is configured via a TOML configuration file. -This file is documented [in the gitaly +This file is documented [in the Gitaly repository](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md). -To change a Gitaly setting in Omnibus you can use -`gitaly['my_setting']` in `/etc/gitlab/gitlab.rb`. Changes will be applied -when you run `gitlab-ctl reconfigure`. +In case you want to change some of its settings: -```ruby -gitaly['prometheus_listen_addr'] = 'localhost:9236' -``` - -To change a Gitaly setting in installations from source you can edit -`/home/git/gitaly/config.toml`. Changes will be applied when you run -`service gitlab restart`. +**For Omnibus GitLab** -```toml -prometheus_listen_addr = "localhost:9236" -``` +1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -## Client-side GRPC logs +**For installations from source** -Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain useful information when -you are seeing Gitaly errors. You can control the log level of the -gRPC client with the `GRPC_LOG_LEVEL` environment variable. The -default level is `WARN`. +1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example). +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). ## Running Gitaly on its own server -> This is an optional way to deploy Gitaly which can benefit GitLab +This is an optional way to deploy Gitaly which can benefit GitLab installations that are larger than a single machine. Most installations will be better served with the default configuration used by Omnibus and the GitLab source installation guide. Starting with GitLab 11.4, Gitaly is able to serve all Git requests without -needed a shared NFS mount for Git repository data. +requiring a shared NFS mount for Git repository data. Between 11.4 and 11.8 the exception was the -[Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +[Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). But since 11.8 the indexer uses Gitaly for data access as well. NFS can still be leveraged for redudancy on block level of the Git data. But only has to be mounted on the Gitaly server. -NOTE: **Note:** While Gitaly can be used as a replacement for NFS, we do not recommend -using EFS as it may impact GitLab's performance. Please review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) +Starting with GitLab 11.8, it is possible to use ElasticSearch in conjunction with +a Gitaly setup that isn't utilising NFS. In order to use ElasticSearch in this +scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer-beta) +needs to be enabled in your GitLab configuration. + +NOTE: **Note:** While Gitaly can be used as a replacement for NFS, it's not recommended +to use EFS as it may impact GitLab's performance. Review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. ### Network architecture -- gitlab-rails shards repositories into "repository storages" -- `gitlab-rails/config/gitlab.yml` contains a map from storage names to - (Gitaly address, Gitaly token) pairs -- the `storage name` -\> `(Gitaly address, Gitaly token)` map in - `gitlab.yml` is the single source of truth for the Gitaly network - topology -- a (Gitaly address, Gitaly token) corresponds to a Gitaly server -- a Gitaly server hosts one or more storages -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients -- Gitaly clients are: unicorn, sidekiq, gitlab-workhorse, - gitlab-shell, Elasticsearch Indexer, and Gitaly itself -- special case: a Gitaly server must be able to make RPC calls **to - itself** via its own (Gitaly address, Gitaly token) pair as - specified in `gitlab-rails/config/gitlab.yml` -- Gitaly servers must not be exposed to the public internet - -Gitaly network traffic is unencrypted by default, but supports -[TLS](#tls-support). Authentication is done through a static token. - -NOTE: **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to -restrict access to your Gitaly server. +The following list depicts what the network architecture of Gitaly is: + +- GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md). +- `/config/gitlab.yml` contains a map from storage names to + `(Gitaly address, Gitaly token)` pairs. +- the `storage name` -\> `(Gitaly address, Gitaly token)` map in + `/config/gitlab.yml` is the single source of truth for the Gitaly network + topology. +- A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server. +- A Gitaly server hosts one or more storages. +- Gitaly addresses must be specified in such a way that they resolve + correctly for ALL Gitaly clients. +- Gitaly clients are: Unicorn, Sidekiq, gitlab-workhorse, + gitlab-shell, Elasticsearch Indexer, and Gitaly itself. +- A Gitaly server must be able to make RPC calls **to itself** via its own + `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. +- Gitaly servers must not be exposed to the public internet as Gitaly's network + traffic is unencrypted by default. The use of firewall is highly recommended + to restrict access to the Gitaly server. Another option is to + [use TLS](#tls-support). +- Authentication is done through a static token which is shared among the Gitaly + and GitLab Rails nodes. Below we describe how to configure a Gitaly server at address `gitaly.internal:8075` with secret token `abc123secret`. We assume your GitLab installation has two repository storages, `default` and `storage1`. -### Installation +### 1. Installation -First install Gitaly using either Omnibus or from source. +First install Gitaly using either Omnibus GitLab or install it from source: -Omnibus: [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab -package you want using **steps 1 and 2** from the GitLab downloads page but -**_do not_** provide the `EXTERNAL_URL=` value. +- For Omnibus GitLab: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page but + **_do not_** provide the `EXTERNAL_URL=` value. +- From source: [Install Gitaly](../../install/installation.md#install-gitaly). -Source: [Install Gitaly](../../install/installation.md#install-gitaly) +### 2. Client side token configuration -### Client side token configuration +Configure a token on the instance that runs the GitLab Rails application. -Configure a token on the client side. +**For Omnibus GitLab** -Omnibus installations: +1. On the client node(s), edit `/etc/gitlab/gitlab.rb`: -```ruby -# /etc/gitlab/gitlab.rb -gitlab_rails['gitaly_token'] = 'abc123secret' -``` + ```ruby + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -Source installations: +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - gitaly: - token: 'abc123secret' -``` +**For installations from source** -You need to reconfigure (Omnibus) or restart (source) for these -changes to be picked up. +1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml`: -### Gitaly server configuration + ```yaml + gitlab: + gitaly: + token: 'abc123secret' + ``` -Next, on the Gitaly server, we need to configure storage paths, enable +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +### 3. Gitaly server configuration + +Next, on the Gitaly server, you need to configure storage paths, enable the network listener and configure the token. NOTE: **Note:** if you want to reduce the risk of downtime when you enable @@ -137,129 +134,194 @@ the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gi from an existing GitLab server to the Gitaly server. Without this shared secret, Git operations in GitLab will result in an API error. -NOTE: **Note:** In most or all cases the storage paths below end in `/repositories` which is -different than `path` in `git_data_dirs` of Omnibus installations. Check the -directory layout on your Gitaly server to be sure. - -Omnibus installations: - -<!-- -updates to following example must also be made at -https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab ---> - -```ruby -# /etc/gitlab/gitlab.rb - -# Avoid running unnecessary services on the Gitaly server -postgresql['enable'] = false -redis['enable'] = false -nginx['enable'] = false -prometheus['enable'] = false -unicorn['enable'] = false -sidekiq['enable'] = false -gitlab_workhorse['enable'] = false - -# Prevent database connections during 'gitlab-ctl reconfigure' -gitlab_rails['rake_cache_clear'] = false -gitlab_rails['auto_migrate'] = false - -# Configure the gitlab-shell API callback URL. Without this, `git push` will -# fail. This can be your 'front door' GitLab URL or an internal load -# balancer. -# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. -gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - -# Make Gitaly accept connections on all network interfaces. You must use -# firewalls to restrict access to this address/port. -gitaly['listen_addr'] = "0.0.0.0:8075" -gitaly['auth_token'] = 'abc123secret' - -gitaly['storage'] = [ - { 'name' => 'default', 'path' => '/mnt/gitlab/default/repositories' }, - { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' }, -] - -# To use TLS for Gitaly you need to add -gitaly['tls_listen_addr'] = "0.0.0.0:9999" -gitaly['certificate_path'] = "path/to/cert.pem" -gitaly['key_path'] = "path/to/key.pem" -``` +NOTE: **Note:** +In most or all cases, the storage paths below end in `/repositories` which is +not that case with `path` in `git_data_dirs` of Omnibus GitLab installations. +Check the directory layout on your Gitaly server to be sure. -Source installations: +**For Omnibus GitLab** -```toml -# /home/git/gitaly/config.toml -listen_addr = '0.0.0.0:8075' -tls_listen_addr = '0.0.0.0:9999' +1. Edit `/etc/gitlab/gitlab.rb`: -[tls] -certificate_path = /path/to/cert.pem -key_path = /path/to/key.pem + <!-- + updates to following example must also be made at + https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> -[auth] -token = 'abc123secret' + ```ruby + # /etc/gitlab/gitlab.rb -[[storage]] -name = 'default' -path = '/mnt/gitlab/default/repositories' + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false -[[storage]] -name = 'storage1' -path = '/mnt/gitlab/storage1/repositories' -``` + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + gitaly['listen_addr'] = "0.0.0.0:8075" + gitaly['auth_token'] = 'abc123secret' + + gitaly['storage'] = [ + { 'name' => 'default' }, + { 'name' => 'storage1' }, + ] + + # To use TLS for Gitaly you need to add + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` + + NOTE: **Note:** + In some cases, you'll have to set `path` for `gitaly['storage']` in the + format `'path' => '/mnt/gitlab/<storage name>/repositories'`. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. On the client node(s), edit `/home/git/gitaly/config.toml`: + + ```toml + listen_addr = '0.0.0.0:8075' + tls_listen_addr = '0.0.0.0:9999' + + [tls] + certificate_path = /path/to/cert.pem + key_path = /path/to/key.pem + + [auth] + token = 'abc123secret' + + [[storage]] + name = 'default' -Again, reconfigure (Omnibus) or restart (source). + [[storage]] + name = 'storage1' + ``` -### Converting clients to use the Gitaly server + NOTE: **Note:** + In some cases, you'll have to set `path` for each `[[storage]]` in the + format `path = '/mnt/gitlab/<storage name>/repositories'`. -Now as the final step update the client machines to switch from using -their local Gitaly service to the new Gitaly server you just -configured. This is a risky step because if there is any sort of -network, firewall, or name resolution problem preventing your GitLab -server from reaching the Gitaly server then all Gitaly requests will -fail. +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -Additionally, you need to +### 4. Converting clients to use the Gitaly server + +As the final step, you need to update the client machines to switch from using +their local Gitaly service to the new Gitaly server you just configured. This +is a risky step because if there is any sort of network, firewall, or name +resolution problem preventing your GitLab server from reaching the Gitaly server, +then all Gitaly requests will fail. + +Additionally, you need to [disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). We assume that your Gitaly server can be reached at `gitaly.internal:8075` from your GitLab server, and that Gitaly can read and write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively. -Omnibus installations: +**For Omnibus GitLab** -```ruby -# /etc/gitlab/gitlab.rb -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, -}) +1. Edit `/etc/gitlab/gitlab.rb`: -gitlab_rails['gitaly_token'] = 'abc123secret' -``` + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, + }) -Source installations: + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - repositories: - storages: - default: - gitaly_address: tcp://gitaly.internal:8075 - storage1: - gitaly_address: tcp://gitaly.internal:8075 + NOTE: **Note:** + In some cases, you'll have to set `path` for each `git_data_dirs` in the + format `'path' => '/mnt/gitlab/<storage name>'`. - gitaly: - token: 'abc123secret' -``` +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Tail the logs to see the requests: + + ```sh + sudo gitlab-ctl tail gitaly + ``` + +**For installations from source** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: tcp://gitaly.internal:8075 + storage1: + gitaly_address: tcp://gitaly.internal:8075 + + gitaly: + token: 'abc123secret' + ``` + + NOTE: **Note:** + In some cases, you'll have to set `path` for each of the `storages` in the + format `path: /mnt/gitlab/<storage name>/repositories`. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Tail the logs to see the requests: + + ```sh + tail -f /home/git/gitlab/log/gitaly.log + ``` + +When you tail the Gitaly logs on your Gitaly server you should see requests +coming in. One sure way to trigger a Gitaly request is to clone a repository +from your GitLab server over HTTP. + +### Disabling the Gitaly service in a cluster environment + +If you are running Gitaly [as a remote +service](#running-gitaly-on-its-own-server) you may want to disable +the local Gitaly service that runs on your GitLab server by default. +Disabling Gitaly only makes sense when you run GitLab in a custom +cluster configuration, where different services run on different +machines. Disabling Gitaly on all machines in the cluster is not a +valid configuration. + +To disable Gitaly on a client node: + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['enable'] = false + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** -Now reconfigure (Omnibus) or restart (source). When you tail the -Gitaly logs on your Gitaly server (`sudo gitlab-ctl tail gitaly` or -`tail -f /home/git/gitlab/log/gitaly.log`) you should see requests -coming in. One sure way to trigger a Gitaly request is to clone a -repository from your GitLab server over HTTP. +1. Edit `/etc/default/gitlab`: + + ```shell + gitaly_enabled=false + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). ## TLS support @@ -267,172 +329,163 @@ repository from your GitLab server over HTTP. Gitaly supports TLS encryption. To be able to communicate with a Gitaly instance that listens for secure connections you will need to use `tls://` url -scheme in the `gitaly_address` of the corresponding storage entry in the gitlab configuration. +scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. -The admin needs to bring their own certificate as we do not provide that automatically. -The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +You will need to bring your own certificates as this isn't provided automatically. +The certificate to be used needs to be installed on all Gitaly nodes and on all +client nodes that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -Note that it is possible to configure Gitaly servers with both an +NOTE: **Note:** +It is possible to configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if necessary. -To observe what type of connections are actually being used in a -production environment you can use the following Prometheus query: +To configure Gitaly with TLS: -``` -sum(rate(gitaly_connections_total[5m])) by (type) -``` +**For Omnibus GitLab** -### Example TLS configuration +1. On the client nodes, edit `/etc/gitlab/gitlab.rb`: -### Omnibus installations: + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly.internal:9999' }, + }) -#### On client nodes: + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -```ruby -# /etc/gitlab/gitlab.rb -git_data_dirs({ - 'default' => { 'path' => '/mnt/gitlab/default', 'gitaly_address' => 'tls://gitaly.internal:9999' }, - 'storage1' => { 'path' => '/mnt/gitlab/storage1', 'gitaly_address' => 'tls://gitaly.internal:9999' }, -}) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly server nodes, edit `/etc/gitlab/gitlab.rb`: -gitlab_rails['gitaly_token'] = 'abc123secret' -``` + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` -#### On Gitaly server nodes: +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -```ruby -gitaly['tls_listen_addr'] = "0.0.0.0:9999" -gitaly['certificate_path'] = "path/to/cert.pem" -gitaly['key_path'] = "path/to/key.pem" -``` +**For installations from source** -### Source installations: +1. On the client nodes, edit `/home/git/gitlab/config/gitlab.yml`: -#### On client nodes: + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: tls://gitaly.internal:9999 + storage1: + gitaly_address: tls://gitaly.internal:9999 -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - repositories: - storages: - default: - path: /mnt/gitlab/default/repositories - gitaly_address: tls://gitaly.internal:9999 - storage1: - path: /mnt/gitlab/storage1/repositories - gitaly_address: tls://gitaly.internal:9999 + gitaly: + token: 'abc123secret' + ``` - gitaly: - token: 'abc123secret' -``` + NOTE: **Note:** + In some cases, you'll have to set `path` for each of the `storages` in the + format `path: /mnt/gitlab/<storage name>/repositories`. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly server nodes, edit `/home/git/gitaly/config.toml`: -#### On Gitaly server nodes: + ```toml + tls_listen_addr = '0.0.0.0:9999' -```toml -# /home/git/gitaly/config.toml -tls_listen_addr = '0.0.0.0:9999' + [tls] + certificate_path = '/path/to/cert.pem' + key_path = '/path/to/key.pem' + ``` -[tls] -certificate_path = '/path/to/cert.pem' -key_path = '/path/to/key.pem' +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +To observe what type of connections are actually being used in a +production environment you can use the following Prometheus query: + +``` +sum(rate(gitaly_connections_total[5m])) by (type) ``` -## Gitaly-ruby +## `gitaly-ruby` -Gitaly was developed to replace Ruby application code in gitlab-ce/ee. +Gitaly was developed to replace the Ruby application code in GitLab. In order to save time and/or avoid the risk of rewriting existing application logic, in some cases we chose to copy some application code -from gitlab-ce into Gitaly almost as-is. To be able to run that code, we -made gitaly-ruby, which is a sidecar process for the main Gitaly Go -process. Some examples of things that are implemented in gitaly-ruby are -RPC's that deal with wiki's, and RPC's that create commits on behalf of +from GitLab into Gitaly almost as-is. To be able to run that code, +`gitaly-ruby` was created, which is a "sidecar" process for the main Gitaly Go +process. Some examples of things that are implemented in `gitaly-ruby` are +RPCs that deal with wikis, and RPCs that create commits on behalf of a user, such as merge commits. -### Number of gitaly-ruby workers +### Number of `gitaly-ruby` workers -Gitaly-ruby has much less capacity than Gitaly itself. If your Gitaly -server has to handle a lot of request, the default setting of having -just 1 active gitaly-ruby sidecar might not be enough. If you see -ResourceExhausted errors from Gitaly it's very likely that you have not -enough gitaly-ruby capacity. +`gitaly-ruby` has much less capacity than Gitaly itself. If your Gitaly +server has to handle a lot of requests, the default setting of having +just one active `gitaly-ruby` sidecar might not be enough. If you see +`ResourceExhausted` errors from Gitaly, it's very likely that you have not +enough `gitaly-ruby` capacity. -You can increase the number of gitaly-ruby processes on your Gitaly +You can increase the number of `gitaly-ruby` processes on your Gitaly server with the following settings. -Omnibus: +**For Omnibus GitLab** -```ruby -# /etc/gitlab/gitlab.rb -# Default is 2 workers. The minimum is 2; 1 worker is always reserved as -# a passive stand-by. -gitaly['ruby_num_workers'] = 4 -``` +1. Edit `/etc/gitlab/gitlab.rb`: -Source: + ```ruby + # Default is 2 workers. The minimum is 2; 1 worker is always reserved as + # a passive stand-by. + gitaly['ruby_num_workers'] = 4 + ``` -```toml -# /home/git/gitaly/config.toml -[gitaly-ruby] -num_workers = 4 -``` +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### Observing gitaly-ruby traffic +**For installations from source** -Gitaly-ruby is a somewhat hidden, internal implementation detail of -Gitaly. There is not that much visibility into what goes on inside -gitaly-ruby processes. +1. Edit `/home/git/gitaly/config.toml`: -If you have Prometheus set up to scrape your Gitaly process, you can see -request rates and error codes for individual RPC's in gitaly-ruby by -querying `grpc_client_handled_total`. Strictly speaking this metric does -not differentiate between gitaly-ruby and other RPC's, but in practice -(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal -calls from the main Gitaly process to one of its gitaly-ruby sidecars. - -Assuming your `grpc_client_handled_total` counter only observes Gitaly, -the following query shows you RPC's are (most likely) internally -implemented as calls to gitaly-ruby. - -``` -sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 -``` + ```toml + [gitaly-ruby] + num_workers = 4 + ``` -## Disabling or enabling the Gitaly service in a cluster environment +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -If you are running Gitaly [as a remote -service](#running-gitaly-on-its-own-server) you may want to disable -the local Gitaly service that runs on your GitLab server by default. +## Eliminating NFS altogether -> 'Disabling Gitaly' only makes sense when you run GitLab in a custom -cluster configuration, where different services run on different -machines. Disabling Gitaly on all machines in the cluster is not a -valid configuration. +If you are planning to use Gitaly without NFS for your storage needs +and want to eliminate NFS from your environment altogether, there are +a few things that you need to do: -If you are setting up a GitLab cluster where Gitaly does not need to -run on all machines, you can disable the Gitaly service in your -Omnibus installation, add the following line to `/etc/gitlab/gitlab.rb`: +1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. +1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md) + to eliminate the need for a shared authorized_keys file. +1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) + including [live tracing](../job_traces.md#new-live-trace-architecture). +1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). +1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). -```ruby -gitaly['enable'] = false -``` +NOTE: **Note:** +One current feature of GitLab that still requires a shared directory (NFS) is +[GitLab Pages](../../user/project/pages/index.md). +There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) +to eliminate the need for NFS to support GitLab Pages. -When you run `gitlab-ctl reconfigure` the Gitaly service will be -disabled. +## Troubleshooting Gitaly -To disable the Gitaly service in a GitLab cluster where you installed -GitLab from source, add the following to `/etc/default/gitlab` on the -machine where you want to disable Gitaly. +### Commits, pushes, and clones return a 401 -```shell -gitaly_enabled=false +``` +remote: GitLab: 401 Unauthorized ``` -When you run `service gitlab restart` Gitaly will be disabled on this -particular machine. +You will need to sync your `gitlab-secrets.json` file with your GitLab +app nodes. -## Troubleshooting Gitaly in production +### `gitaly-debug` Since GitLab 11.6, Gitaly comes with a command-line tool called `gitaly-debug` that can be run on a Gitaly server to aid in @@ -442,3 +495,123 @@ Git clone speed for a specific repository on the server. For an up to date list of sub-commands see [the gitaly-debug README](https://gitlab.com/gitlab-org/gitaly/blob/master/cmd/gitaly-debug/README.md). + +### Client side GRPC logs + +Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC +client has its own log file which may contain useful information when +you are seeing Gitaly errors. You can control the log level of the +gRPC client with the `GRPC_LOG_LEVEL` environment variable. The +default level is `WARN`. + +### Observing `gitaly-ruby` traffic + +[`gitaly-ruby`](#gitaly-ruby) is an internal implementation detail of Gitaly, +so, there's not that much visibility into what goes on inside +`gitaly-ruby` processes. + +If you have Prometheus set up to scrape your Gitaly process, you can see +request rates and error codes for individual RPCs in `gitaly-ruby` by +querying `grpc_client_handled_total`. Strictly speaking, this metric does +not differentiate between `gitaly-ruby` and other RPCs, but in practice +(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal +calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. + +Assuming your `grpc_client_handled_total` counter only observes Gitaly, +the following query shows you RPCs are (most likely) internally +implemented as calls to `gitaly-ruby`: + +``` +sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 +``` + +### Repository changes fail with a `401 Unauthorized` error + +If you're running Gitaly on its own server and notice that users can +successfully clone and fetch repositories (via both SSH and HTTPS), but can't +push to them or make changes to the repository in the web UI without getting a +`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate +with the other nodes due to having the [wrong secrets file](#3-gitaly-server-configuration). + +Confirm the following are all true: + +- When any user performs a `git push` to any repository on this Gitaly node, it + fails with the following error (note the `401 Unauthorized`): + + ```sh + remote: GitLab: 401 Unauthorized + To <REMOTE_URL> + ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) + error: failed to push some refs to '<REMOTE_URL>' + ``` + +- When any user adds or modifies a file from the repository using the GitLab + UI, it immediatley fails with a red `401 Unauthorized` banner. +- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) + successfully creates the project but doesn't create the README. +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.md#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors + when reaching the `/api/v4/internal/allowed` endpoint: + + ```sh + # api_json.log + { + "time": "2019-07-18T00:30:14.967Z", + "severity": "INFO", + "duration": 0.57, + "db": 0, + "view": 0.57, + "status": 401, + "method": "POST", + "path": "\/api\/v4\/internal\/allowed", + "params": [ + { + "key": "action", + "value": "git-receive-pack" + }, + { + "key": "changes", + "value": "REDACTED" + }, + { + "key": "gl_repository", + "value": "REDACTED" + }, + { + "key": "project", + "value": "\/path\/to\/project.git" + }, + { + "key": "protocol", + "value": "web" + }, + { + "key": "env", + "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" + }, + { + "key": "user_id", + "value": "2" + }, + { + "key": "secret_token", + "value": "[FILTERED]" + } + ], + "host": "gitlab.example.com", + "ip": "REDACTED", + "ua": "Ruby", + "route": "\/api\/:version\/internal\/allowed", + "queue_duration": 4.24, + "gitaly_calls": 0, + "gitaly_duration": 0, + "correlation_id": "XPUZqTukaP3" + } + + # nginx_access.log + [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" + ``` + +To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-server-configuration) +on the Gitaly node matches the one on all other nodes. If it doesn't match, +update the secrets file on the Gitaly node to match the others, then +[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index d9c80b1ec59..56665ba8b9a 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -1,3 +1,7 @@ +--- +type: reference, concepts +--- + # Scaling and High Availability GitLab supports several different types of clustering and high-availability. @@ -65,6 +69,7 @@ larger one. - 1 Redis node - 1 NFS/Gitaly storage server - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq) +- 1 Monitoring node (Prometheus, Grafana) #### Installation Instructions @@ -76,6 +81,7 @@ you can continue with the next step. 1. [Redis](redis.md#redis-in-a-scaled-environment) 1. [Gitaly](gitaly.md) (recommended) or [NFS](nfs.md) 1. [GitLab application nodes](gitlab.md) +1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) ### Full Scaling @@ -90,6 +96,7 @@ in size, indicating that there is contention or not enough resources. - 2 or more NFS/Gitaly storage servers - 2 or more Sidekiq nodes - 2 or more GitLab application nodes (Unicorn, Workhorse) +- 1 Monitoring node (Prometheus, Grafana) ## High Availability Architecture Examples @@ -133,6 +140,7 @@ the contention. - 3 Consul/Sentinel nodes - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer) - 1 NFS/Gitaly server +- 1 Monitoring node (Prometheus, Grafana)  @@ -156,25 +164,24 @@ contention due to certain workloads. #### Reference Architecture -- **Status:** Work-in-progress - **Supported Users (approximate):** 10,000 -- **Related Issues:** [gitlab-com/support/support-team-meta#1513](https://gitlab.com/gitlab-com/support/support-team-meta/issues/1513), - [gitlab-org/quality/team-tasks#110](https://gitlab.com/gitlab-org/quality/team-tasks/issues/110) - -The Support and Quality teams are in the process of building and performance testing -an environment that will support about 10,000 users. The specifications below -are a work-in-progress representation of the work so far. Quality will be -certifying this environment in FY20-Q2. The specifications may be adjusted -prior to certification based on performance testing. - -- 3 PostgreSQL - 4 CPU, 8GB RAM -- 1 PgBouncer - 2 CPU, 4GB RAM -- 2 Redis - 2 CPU, 8GB RAM -- 3 Consul/Sentinel - 2 CPU, 2GB RAM -- 4 Sidekiq - 4 CPU, 8GB RAM -- 5 GitLab application nodes - 20 CPU, 64GB RAM -- 1 Gitaly - 20 CPU, 64GB RAM -- 1 Monitoring node - 4 CPU, 8GB RAM +- **Known Issues:** While validating the reference architecture, slow endpoints were discovered and are being investigated. [gitlab-org/gitlab-ce/issues/64335](https://gitlab.com/gitlab-org/gitlab-ce/issues/64335) + +The Support and Quality teams built, performance tested, and validated an +environment that supports about 10,000 users. The specifications below are a +representation of the work so far. The specifications may be adjusted in the +future based on additional testing and iteration. + +NOTE: **Note:** The specifications here were performance tested against a specific coded workload. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + +- 3 PostgreSQL - 4 CPU, 16GiB memory per node +- 1 PgBouncer - 2 CPU, 4GiB memory +- 2 Redis - 2 CPU, 8GiB memory per node +- 3 Consul/Sentinel - 2 CPU, 2GiB memory per node +- 4 Sidekiq - 4 CPU, 16GiB memory per node +- 5 GitLab application nodes - 16 CPU, 64GiB memory per node +- 1 Gitaly - 16 CPU, 64GiB memory +- 1 Monitoring node - 2 CPU, 8GiB memory, 100GiB local storage ### Fully Distributed @@ -192,6 +199,7 @@ with the added complexity of many more nodes to configure, manage and monitor. - 2 or more API nodes (All requests to `/api`) - 2 or more Web nodes (All other web requests) - 2 or more NFS/Gitaly servers +- 1 Monitoring node (Prometheus, Grafana)  @@ -205,4 +213,4 @@ separately: 1. [NFS Client and Host setup](nfs_host_client_setup.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) - +1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) diff --git a/doc/administration/high_availability/alpha_database.md b/doc/administration/high_availability/alpha_database.md index 7bf20be60e6..7afd739f44c 100644 --- a/doc/administration/high_availability/alpha_database.md +++ b/doc/administration/high_availability/alpha_database.md @@ -2,5 +2,4 @@ redirect_to: 'database.md' --- -This documentation has been moved to the main -[database documentation](database.md#configure_using_omnibus_for_high_availability). +This document was moved to [another location](database.md). diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md index 056b7fc15d9..b02a61b9256 100644 --- a/doc/administration/high_availability/consul.md +++ b/doc/administration/high_availability/consul.md @@ -1,16 +1,85 @@ -# Working with the bundled Consul service **[PREMIUM ONLY]** +--- +type: reference +--- -## Overview +# Working with the bundled Consul service **(PREMIUM ONLY)** -As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](http://consul.io) that can be managed through `/etc/gitlab/gitlab.rb`. +As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](https://www.consul.io/) that can be managed through `/etc/gitlab/gitlab.rb`. A Consul cluster consists of multiple server agents, as well as client agents that run on other nodes which need to talk to the consul cluster. +## Prerequisites + +First, make sure to [download/install](https://about.gitlab.com/install/) +GitLab Omnibus **on each node**. + +Choose an installation method, then make sure you complete steps: + +1. Install and configure the necessary dependencies. +1. Add the GitLab package repository and install the package. + +When installing the GitLab package, do not supply `EXTERNAL_URL` value. + +## Configuring the Consul nodes + +On each Consul node perform the following: + +1. Make sure you collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step, before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Consul + roles ['consul_role'] + + # START user configuration + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + server: true, + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + # + # END user configuration + ``` + + > `consul_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes + to take effect. + +### Consul checkpoint + +Before moving on, make sure Consul is configured correctly. Run the following +command to verify all server nodes are communicating: + +```sh +/opt/gitlab/embedded/bin/consul members +``` + +The output should be similar to: + +``` +Node Address Status Type Build Protocol DC +CONSUL_NODE_ONE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +CONSUL_NODE_TWO XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +CONSUL_NODE_THREE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +``` + +If any of the nodes isn't `alive` or if any of the three nodes are missing, +check the [Troubleshooting section](#troubleshooting) before proceeding. + ## Operations ### Checking cluster membership To see which nodes are part of the cluster, run the following on any member in the cluster + ``` # /opt/gitlab/embedded/bin/consul members Node Address Status Type Build Protocol DC @@ -46,18 +115,18 @@ You will see messages like the following in `gitlab-ctl tail consul` output if y 2017-09-25_19:53:41.74356 2017/09/25 19:53:41 [ERR] agent: failed to sync remote state: No cluster leader ``` - To fix this: 1. Pick an address on each node that all of the other nodes can reach this node through. 1. Update your `/etc/gitlab/gitlab.rb` - ```ruby - consul['configuration'] = { - ... - bind_addr: 'IP ADDRESS' - } - ``` + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` + 1. Run `gitlab-ctl reconfigure` If you still see the errors, you may have to [erase the consul database and reinitialize](#recreate-from-scratch) on the affected node. @@ -78,12 +147,13 @@ To fix this: 1. Pick an address on the node that all of the other nodes can reach this node through. 1. Update your `/etc/gitlab/gitlab.rb` - ```ruby - consul['configuration'] = { - ... - bind_addr: 'IP ADDRESS' - } - ``` + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` + 1. Run `gitlab-ctl reconfigure` ### Outage recovery @@ -91,6 +161,7 @@ To fix this: If you lost enough server agents in the cluster to break quorum, then the cluster is considered failed, and it will not function without manual intervenetion. #### Recreate from scratch + By default, GitLab does not store anything in the consul cluster that cannot be recreated. To erase the consul database and reinitialize ``` @@ -102,4 +173,5 @@ By default, GitLab does not store anything in the consul cluster that cannot be After this, the cluster should start back up, and the server agents rejoin. Shortly after that, the client agents should rejoin as well. #### Recover a failed cluster + If you have taken advantage of consul to store other data, and want to restore the failed cluster, please follow the [Consul guide](https://www.consul.io/docs/guides/outage.html) to recover a failed cluster. diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index 3b874e5d312..7c9e02d889e 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -1,6 +1,13 @@ +--- +type: reference +--- + # Configuring PostgreSQL for Scaling and High Availability -## Provide your own PostgreSQL instance **[CORE ONLY]** +In this section, you'll be guided through configuring a PostgreSQL database +to be used with GitLab in a highly available environment. + +## Provide your own PostgreSQL instance **(CORE ONLY)** If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. For example, AWS offers a managed Relational @@ -21,27 +28,27 @@ This section is relevant for [Scaled Architecture](README.md#scalable-architectu environments including [Basic Scaling](README.md#basic-scaling) and [Full Scaling](README.md#full-scaling). -### Provide your own PostgreSQL instance **[CORE ONLY]** +### Provide your own PostgreSQL instance **(CORE ONLY)** If you want to use your own deployed PostgreSQL instance(s), see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) for more details. However, you can use the GitLab Omnibus package to easily deploy the bundled PostgreSQL. -### Standalone PostgreSQL using GitLab Omnibus **[CORE ONLY]** +### Standalone PostgreSQL using GitLab Omnibus **(CORE ONLY)** 1. SSH into the PostgreSQL server. -1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - - Do not complete any other steps on the download page. + - Do not complete any other steps on the download page. 1. Generate a password hash for PostgreSQL. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next step as the value of `POSTGRESQL_PASSWORD_HASH`. - ```sh - sudo gitlab-ctl pg-password-md5 gitlab - ``` + ```sh + sudo gitlab-ctl pg-password-md5 gitlab + ``` 1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder values appropriately. @@ -51,37 +58,38 @@ deploy the bundled PostgreSQL. addresses of the GitLab application servers that will connect to the database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` - ```ruby - # Disable all components except PostgreSQL - roles ['postgres_role'] - repmgr['enable'] = false - consul['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - redis_exporter['enable'] = false - gitlab_monitor['enable'] = false - - postgresql['listen_address'] = '0.0.0.0' - postgresql['port'] = 5432 - - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - # ???? - postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - ``` + ```ruby + # Disable all components except PostgreSQL + roles ['postgres_role'] + repmgr['enable'] = false + consul['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_monitor['enable'] = false + + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + # ???? + postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + ``` - NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 + NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 1. [Reconfigure GitLab] for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and plain text password. These will be necessary when configuring the GitLab application servers later. +1. [Enable monitoring](#enable-monitoring) Advanced configuration options are supported and can be added if needed. @@ -96,14 +104,14 @@ environments including [Horizontal](README.md#horizontal), [Hybrid](README.md#hybrid), and [Fully Distributed](README.md#fully-distributed). -### Provide your own PostgreSQL instance **[CORE ONLY]** +### Provide your own PostgreSQL instance **(CORE ONLY)** If you want to use your own deployed PostgreSQL instance(s), see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) for more details. However, you can use the GitLab Omnibus package to easily deploy the bundled PostgreSQL. -### High Availability with GitLab Omnibus **[PREMIUM ONLY]** +### High Availability with GitLab Omnibus **(PREMIUM ONLY)** > Important notes: > @@ -111,7 +119,7 @@ deploy the bundled PostgreSQL. > - If you are a Community Edition or Starter user, consider using a cloud hosted solution. > - This document will not cover installations from source. > -> - If HA setup is not what you were looking for, see the [database configuration document](http://docs.gitlab.com/omnibus/settings/database.html) +> - If HA setup is not what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) > for the Omnibus GitLab packages. > > Please read this document fully before attempting to configure PostgreSQL HA @@ -193,9 +201,9 @@ When using default setup, minimum configuration requires: - `CONSUL_PASSWORD_HASH`. This is a hash generated out of consul username/password pair. Can be generated with: - ```sh - sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME - ``` + ```sh + sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME + ``` - `CONSUL_SERVER_NODES`. The IP addresses or DNS records of the Consul server nodes. @@ -203,9 +211,9 @@ Few notes on the service itself: - The service runs under a system account, by default `gitlab-consul`. - If you are using a different username, you will have to specify it. We -will refer to it with `CONSUL_USERNAME`, + will refer to it with `CONSUL_USERNAME`, - There will be a database user created with read only access to the repmgr -database + database - Passwords will be stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed @@ -236,9 +244,9 @@ We will need the following password information for the application's database u - `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. Can be generated with: - ```sh - sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME - ``` + ```sh + sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME + ``` ##### Pgbouncer information @@ -249,9 +257,9 @@ When using default setup, minimum configuration requires: - `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of pgbouncer username/password pair. Can be generated with: - ```sh - sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME - ``` + ```sh + sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME + ``` - `PGBOUNCER_NODE`, is the IP address or a FQDN of the node running Pgbouncer. @@ -280,125 +288,76 @@ Few notes on the service itself: #### Installing Omnibus GitLab -First, make sure to [download/install](https://about.gitlab.com/installation) +First, make sure to [download/install](https://about.gitlab.com/install/) GitLab Omnibus **on each node**. Make sure you install the necessary dependencies from step 1, add GitLab package repository from step 2. When installing the GitLab package, do not supply `EXTERNAL_URL` value. -#### Configuring the Consul nodes - -On each Consul node perform the following: - -1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information) before executing the next step. - -1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - - ```ruby - # Disable all components except Consul - roles ['consul_role'] - - # START user configuration - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - server: true, - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - - # Disable auto migrations - gitlab_rails['auto_migrate'] = false - # - # END user configuration - ``` - - > `consul_role` was introduced with GitLab 10.3 - -1. [Reconfigure GitLab] for the changes to take effect. - -##### Consul Checkpoint - -Before moving on, make sure Consul is configured correctly. Run the following -command to verify all server nodes are communicating: - -```sh -/opt/gitlab/embedded/bin/consul members -``` - -The output should be similar to: - -``` -Node Address Status Type Build Protocol DC -CONSUL_NODE_ONE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul -CONSUL_NODE_TWO XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul -CONSUL_NODE_THREE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul -``` - -If any of the nodes isn't `alive` or if any of the three nodes are missing, -check the [Troubleshooting section](#troubleshooting) before proceeding. - #### Configuring the Database nodes +1. Make sure to [configure the Consul nodes](consul.md). 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step. 1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the consul agent - consul['services'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - # Replace X with value of number of db nodes + 1 - postgresql['max_wal_senders'] = X - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) - - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - > `postgres_role` was introduced with GitLab 10.3 + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + # Replace X with value of number of db nodes + 1 + postgresql['max_wal_senders'] = X + postgresql['max_replication_slots'] = X + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) + + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + > `postgres_role` was introduced with GitLab 10.3 1. On secondary nodes, add all the configuration specified above for primary node to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration to inform gitlab-ctl that they are standby nodes initially and it need not attempt to register them as primary node + ``` # HA setting to specify if a node should attempt to be master on initialization repmgr['master_on_initialization'] = false ``` 1. [Reconfigure GitLab] for the changes to take effect. +1. [Enable Monitoring](#enable-monitoring) > Please note: > @@ -416,31 +375,31 @@ Select one node as a primary node. 1. Open a database prompt: - ```sh - gitlab-psql -d gitlabhq_production - ``` + ```sh + gitlab-psql -d gitlabhq_production + ``` 1. Enable the `pg_trgm` extension: - ```sh - CREATE EXTENSION pg_trgm; - ``` + ```sh + CREATE EXTENSION pg_trgm; + ``` 1. Exit the database prompt by typing `\q` and Enter. 1. Verify the cluster is initialized with one node: - ```sh - gitlab-ctl repmgr cluster show - ``` + ```sh + gitlab-ctl repmgr cluster show + ``` - The output should be similar to the following: + The output should be similar to the following: - ``` - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` + ``` + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` 1. Note down the hostname/ip in the connection string: `host=HOSTNAME`. We will refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value @@ -451,43 +410,43 @@ Select one node as a primary node. 1. Set up the repmgr standby: - ```sh - gitlab-ctl repmgr standby setup MASTER_NODE_NAME - ``` - - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - # gitlab-ctl repmgr standby setup MASTER_NODE_NAME - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` + ```sh + gitlab-ctl repmgr standby setup MASTER_NODE_NAME + ``` + + Do note that this will remove the existing data on the node. The command + has a wait time. + + The output should be similar to the following: + + ```console + # gitlab-ctl repmgr standby setup MASTER_NODE_NAME + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s + ``` 1. Verify the node now appears in the cluster: - ```sh - gitlab-ctl repmgr cluster show - ``` + ```sh + gitlab-ctl repmgr cluster show + ``` - The output should be similar to the following: + The output should be similar to the following: - ``` - Role | Name | Upstream | Connection String - ----------+---------|-----------|------------------------------------------------ - * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` + ``` + Role | Name | Upstream | Connection String + ----------+---------|-----------|------------------------------------------------ + * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` Repeat the above steps on all secondary nodes. @@ -527,88 +486,7 @@ Check the [Troubleshooting section](#troubleshooting) before proceeding. #### Configuring the Pgbouncer node -1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. - -1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - - ```ruby - # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] - - # Configure Pgbouncer - pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - - # Configure Consul agent - consul['watchers'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # Replace CONSUL_PASSWORD_HASH with with a generated md5 value - # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value - pgbouncer['users'] = { - 'gitlab-consul': { - password: 'CONSUL_PASSWORD_HASH' - }, - 'pgbouncer': { - password: 'PGBOUNCER_PASSWORD_HASH' - } - } - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - > `pgbouncer_role` was introduced with GitLab 10.3 - -1. [Reconfigure GitLab] for the changes to take effect. - -1. Create a `.pgpass` file so Consule is able to - reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: - - ```sh - gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul - ``` - -##### PGBouncer Checkpoint - -1. Ensure the node is talking to the current master: - - ```sh - gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD - ``` - - If there is an error `psql: ERROR: Auth failed` after typing in the - password, ensure you previously generated the MD5 password hashes with the correct - format. The correct format is to concatenate the password and the username: - `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text - needed to generate an MD5 password hash for the `pgbouncer` user. - -1. Once the console prompt is available, run the following queries: - - ```sh - show databases ; show clients ; - ``` - - The output should be similar to the following: - - ``` - name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections - ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- - gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 - pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 - (2 rows) - - type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls - ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- - C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | - (2 rows) - ``` +See our [documentation for Pgbouncer](pgbouncer.md) for information on running Pgbouncer as part of an HA setup. #### Configuring the Application nodes @@ -617,15 +495,15 @@ attributes set, but the following need to be set. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Disable PostgreSQL on the application node - postgresql['enable'] = false + ```ruby + # Disable PostgreSQL on the application node + postgresql['enable'] = false - gitlab_rails['db_host'] = 'PGBOUNCER_NODE' - gitlab_rails['db_port'] = 6432 - gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' - gitlab_rails['auto_migrate'] = false - ``` + gitlab_rails['db_host'] = 'PGBOUNCER_NODE' + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' + gitlab_rails['auto_migrate'] = false + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -785,45 +663,45 @@ After deploying the configuration follow these steps: 1. On `10.6.0.21`, our primary database - Enable the `pg_trgm` extension + Enable the `pg_trgm` extension - ```sh - gitlab-psql -d gitlabhq_production - ``` + ```sh + gitlab-psql -d gitlabhq_production + ``` - ``` - CREATE EXTENSION pg_trgm; - ``` + ``` + CREATE EXTENSION pg_trgm; + ``` 1. On `10.6.0.22`, our first standby database - Make this node a standby of the primary + Make this node a standby of the primary - ```sh - gitlab-ctl repmgr standby setup 10.6.0.21 - ``` + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` 1. On `10.6.0.23`, our second standby database - Make this node a standby of the primary + Make this node a standby of the primary - ```sh - gitlab-ctl repmgr standby setup 10.6.0.21 - ``` + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` 1. On `10.6.0.31`, our application server - Set gitlab-consul's pgbouncer password to `toomanysecrets` + Set gitlab-consul's pgbouncer password to `toomanysecrets` - ```sh - gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul - ``` + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` - Run database migrations + Run database migrations - ```sh - gitlab-rake gitlab:db:configure - ``` + ```sh + gitlab-rake gitlab:db:configure + ``` #### Example minimal setup @@ -960,16 +838,16 @@ standby nodes. 1. Ensure the old master node is not still active. 1. Login to the server that should become the new master and run: - ```sh - gitlab-ctl repmgr standby promote - ``` + ```sh + gitlab-ctl repmgr standby promote + ``` 1. If there are any other standby servers in the cluster, have them follow the new master server: - ```sh - gitlab-ctl repmgr standby follow NEW_MASTER - ``` + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + ``` #### Restore procedure @@ -979,42 +857,42 @@ after it has been restored to service. - If you want to remove the node from the cluster, on any other node in the cluster, run: - ```sh - gitlab-ctl repmgr standby unregister --node=X - ``` + ```sh + gitlab-ctl repmgr standby unregister --node=X + ``` - where X is the value of node in `repmgr.conf` on the old server. + where X is the value of node in `repmgr.conf` on the old server. - To find this, you can use: + To find this, you can use: - ```sh - awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf - ``` + ```sh + awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf + ``` - It will output something like: + It will output something like: - ``` - 959789412 - ``` + ``` + 959789412 + ``` - Then you will use this id to unregister the node: + Then you will use this id to unregister the node: - ```sh - gitlab-ctl repmgr standby unregister --node=959789412 - ``` + ```sh + gitlab-ctl repmgr standby unregister --node=959789412 + ``` - To add the node as a standby server: - ```sh - gitlab-ctl repmgr standby follow NEW_MASTER - gitlab-ctl restart repmgrd - ``` + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + gitlab-ctl restart repmgrd + ``` - CAUTION: **Warning:** When the server is brought back online, and before - you switch it to a standby node, repmgr will report that there are two masters. - If there are any clients that are still attempting to write to the old master, - this will cause a split, and the old master will need to be resynced from - scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. + CAUTION: **Warning:** When the server is brought back online, and before + you switch it to a standby node, repmgr will report that there are two masters. + If there are any clients that are still attempting to write to the old master, + this will cause a split, and the old master will need to be resynced from + scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. #### Alternate configurations @@ -1057,13 +935,13 @@ the previous section: 1. On the current master node, create a password for the `gitlab` and `gitlab_repmgr` user: - ```sh - gitlab-psql -d template1 - template1=# \password gitlab_repmgr - Enter password: **** - Confirm password: **** - template1=# \password gitlab - ``` + ```sh + gitlab-psql -d template1 + template1=# \password gitlab_repmgr + Enter password: **** + Confirm password: **** + template1=# \password gitlab + ``` 1. On each database node: @@ -1077,18 +955,37 @@ the previous section: 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to when asked: - ```sh - gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' - ``` + ```sh + gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' + ``` 1. On each pgbouncer node, edit `/etc/gitlab/gitlab.rb`: 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for the `gitlab` database user 1. [Reconfigure GitLab] for the changes to take effect +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** database servers. + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ## Troubleshooting -#### Consul and PostgreSQL changes not taking effect. +### Consul and PostgreSQL changes not taking effect Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. @@ -1098,7 +995,7 @@ For PostgreSQL, it is usually safe to restart the master node by default. Automa On the consul server nodes, it is important to restart the consul service in a controlled fashion. Read our [consul documentation](consul.md#restarting-the-server-cluster) for instructions on how to restart the service. -#### `gitlab-ctl repmgr-check-master` command produces errors +### `gitlab-ctl repmgr-check-master` command produces errors If this command displays errors about database permissions it is likely that something failed during install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these @@ -1113,7 +1010,7 @@ steps to fix the problem: Now there should not be errors. If errors still occur then there is another problem. -#### PGBouncer error `ERROR: pgbouncer cannot connect to server` +### PGBouncer error `ERROR: pgbouncer cannot connect to server` You may get this error when running `gitlab-rake gitlab:db:configure` or you may see the error in the PGBouncer log file. @@ -1141,12 +1038,12 @@ postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) [Reconfigure GitLab] for the changes to take effect. -#### Issues with other components +### Issues with other components If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page. - [Consul](consul.md#troubleshooting) -- [PostgreSQL](http://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) +- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) - [GitLab application](gitlab.md#troubleshooting) ## Configure using Omnibus diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index 1d8e6c999cb..739d1ae35fb 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -1,14 +1,18 @@ +--- +type: reference +--- + # Configuring Gitaly for Scaled and High Availability Gitaly does not yet support full high availability. However, Gitaly is quite stable and is in use on GitLab.com. Scaled and highly available GitLab environments -should consider using Gitaly on a separate node. +should consider using Gitaly on a separate node. -See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to -track plans and progress toward high availability support. +See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to +track plans and progress toward high availability support. This document is relevant for [Scaled Architecture](README.md#scalable-architecture-examples) -environments and [High Availability Architecture](README.md#high-availability-architecture-examples). +environments and [High Availability Architecture](README.md#high-availability-architecture-examples). ## Running Gitaly on its own server @@ -19,3 +23,42 @@ Continue configuration of other components by going back to: - [Scaled Architectures](README.md#scalable-architecture-examples) - [High Availability Architectures](README.md#high-availability-architecture-examples) + +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 888426ece5c..8818a9606de 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -1,4 +1,8 @@ -# Configuring GitLab Scaling and High Availability +--- +type: reference +--- + +# Configuring GitLab for Scaling and High Availability > **Note:** There is some additional configuration near the bottom for additional GitLab application servers. It's important to read and understand @@ -7,33 +11,33 @@ 1. If necessary, install the NFS client utility packages using the following commands: - ``` - # Ubuntu/Debian - apt-get install nfs-common + ``` + # Ubuntu/Debian + apt-get install nfs-common - # CentOS/Red Hat - yum install nfs-utils nfs-utils-lib - ``` + # CentOS/Red Hat + yum install nfs-utils nfs-utils-lib + ``` 1. Specify the necessary NFS shares. Mounts are specified in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose to configure your NFS server. See [NFS documentation](nfs.md) for the various options. Here is an example snippet to add to `/etc/fstab`: - ``` - 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - ``` + ``` + 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + ``` 1. Create the shared directories. These may be different depending on your NFS mount locations. - ``` - mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data - ``` + ``` + mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data + ``` 1. Download/install GitLab Omnibus using **steps 1 and 2** from [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other @@ -46,49 +50,52 @@ added NFS mounts in the default data locations. Additionally the UID and GIDs given are just examples and you should configure with your preferred values. - ```ruby - external_url 'https://gitlab.example.com' - - # Prevent GitLab from starting if NFS data mounts are not available - high_availability['mountpoint'] = '/var/opt/gitlab/git-data' - - # Disable components that will not be on the GitLab application server - roles ['application_role'] - nginx['enable'] = true - - # PostgreSQL connection details - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' - gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server - gitlab_rails['db_password'] = 'DB password' - - # Redis connection details - gitlab_rails['redis_port'] = '6379' - gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server - gitlab_rails['redis_password'] = 'Redis Password' - - # Ensure UIDs and GIDs match between servers for permissions via NFS - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` - - > **Note:** To maintain uniformity of links across HA clusters, the `external_url` - on the first application server as well as the additional application - servers should point to the external url that users will use to access GitLab. - In a typical HA setup, this will be the url of the load balancer which will - route traffic to all GitLab application servers in the HA cluster. - > - > **Note:** When you specify `https` in the `external_url`, as in the example - above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If - certificates are not present, Nginx will fail to start. See - [Nginx documentation](http://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) - for more information. - > - > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. + ```ruby + external_url 'https://gitlab.example.com' + + # Prevent GitLab from starting if NFS data mounts are not available + high_availability['mountpoint'] = '/var/opt/gitlab/git-data' + + # Disable components that will not be on the GitLab application server + roles ['application_role'] + nginx['enable'] = true + + # PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + + # Redis connection details + gitlab_rails['redis_port'] = '6379' + gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server + gitlab_rails['redis_password'] = 'Redis Password' + + # Ensure UIDs and GIDs match between servers for permissions via NFS + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +1. [Enable monitoring](#enable-monitoring) + + > **Note:** To maintain uniformity of links across HA clusters, the `external_url` + on the first application server as well as the additional application + servers should point to the external url that users will use to access GitLab. + In a typical HA setup, this will be the url of the load balancer which will + route traffic to all GitLab application servers in the HA cluster. + > + > **Note:** When you specify `https` in the `external_url`, as in the example + above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If + certificates are not present, Nginx will fail to start. See + [Nginx documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) + for more information. + > + > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure + of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. ## First GitLab application server @@ -111,24 +118,65 @@ need some extra configuration. secondary servers **prior to** running the first `reconfigure` in the steps above. - ```ruby - gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860' - gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' - gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' - gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964' - ``` + ```ruby + gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860' + gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' + gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' + gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964' + ``` 1. Run `touch /etc/gitlab/skip-auto-reconfigure` to prevent database migrations from running on upgrade. Only the primary GitLab application server should handle migrations. -1. **Optional** Configure host keys. Copy all contents(primary and public keys) inside `/etc/ssh/` on +1. **Recommended** Configure host keys. Copy the contents (primary and public keys) of `/etc/ssh/` on the primary application server to `/etc/ssh` on all secondary servers. This prevents false man-in-the-middle-attack alerts when accessing servers in your High Availability cluster behind a load balancer. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** GitLab servers. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + sidekiq['listen_address'] = "0.0.0.0" + unicorn['listen'] = '0.0.0.0' + + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with + # the address and/or subnets gathered from the monitoring node(s). + gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +> **Warning:** After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`, + it can take an extended period of time for unicorn to complete reloading after receiving a `HUP`. + For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401). + ## Troubleshooting - `mount: wrong fs type, bad option, bad superblock on` diff --git a/doc/administration/high_availability/img/fully-distributed.png b/doc/administration/high_availability/img/fully-distributed.png Binary files differindex ad23207134e..c3cd2bf24f0 100644 --- a/doc/administration/high_availability/img/fully-distributed.png +++ b/doc/administration/high_availability/img/fully-distributed.png diff --git a/doc/administration/high_availability/img/horizontal.png b/doc/administration/high_availability/img/horizontal.png Binary files differindex c3bd489d96f..75d08e1097a 100644 --- a/doc/administration/high_availability/img/horizontal.png +++ b/doc/administration/high_availability/img/horizontal.png diff --git a/doc/administration/high_availability/img/hybrid.png b/doc/administration/high_availability/img/hybrid.png Binary files differindex 7d4a56bf0ea..8dd9923e597 100644 --- a/doc/administration/high_availability/img/hybrid.png +++ b/doc/administration/high_availability/img/hybrid.png diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md index 28b226cacd5..9e9f604317a 100644 --- a/doc/administration/high_availability/load_balancer.md +++ b/doc/administration/high_availability/load_balancer.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Load Balancer for GitLab HA In an active/active GitLab configuration, you will need a load balancer to route @@ -114,3 +118,15 @@ Read more on high-availability configuration: if SSL was terminated at the load balancer. [gitlab-pages]: ../pages/index.md + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md new file mode 100644 index 00000000000..b2750603c74 --- /dev/null +++ b/doc/administration/high_availability/monitoring_node.md @@ -0,0 +1,87 @@ +--- +type: reference +--- + +# Configuring a Monitoring node for Scaling and High Availability + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +You can configure a Prometheus node to monitor GitLab. + +## Standalone Monitoring node using GitLab Omnibus + +The GitLab Omnibus package can be used to configure a standalone Monitoring node running [Prometheus](../monitoring/prometheus/index.md) and [Grafana](../monitoring/performance/grafana_configuration.md). +The monitoring node is not highly available. See [Scaling and High Availability](README.md) +for an overview of GitLab scaling and high availability options. + +The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with +Omnibus: + +1. SSH into the Monitoring node. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = 'toomanysecrets' + + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +## Migrating to Service Discovery + +Once monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, +ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both +`consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` +will result in errors. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 3013ce63bd5..274bd32299b 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # NFS You can view information and options set for each of the mounted NFS file @@ -42,34 +46,20 @@ maintaining ID mapping without LDAP, in most cases you should enable numeric UID and GIDs (which is off by default in some cases) for simplified permission management between systems: - - [NetApp instructions](https://library.netapp.com/ecmdocs/ECMP1401220/html/GUID-24367A9F-E17B-4725-ADC1-02D86F56F78E.html) - - For non-NetApp devices, disable NFSv4 `idmapping` by performing opposite of [enable NFSv4 idmapper](https://wiki.archlinux.org/index.php/NFS#Enabling_NFSv4_idmapping) +- [NetApp instructions](https://library.netapp.com/ecmdocs/ECMP1401220/html/GUID-24367A9F-E17B-4725-ADC1-02D86F56F78E.html) +- For non-NetApp devices, disable NFSv4 `idmapping` by performing opposite of [enable NFSv4 idmapper](https://wiki.archlinux.org/index.php/NFS#Enabling_NFSv4_idmapping) ### Improving NFS performance with GitLab -NOTE: **Note:** This is only available starting in certain versions of GitLab: 11.5.11, -11.6.11, 11.7.12, 11.8.8, 11.9.0 and up (e.g. 11.10, 11.11, etc.) +NOTE: **Note:** From GitLab 12.1, it will automatically be detected if Rugged can and should be used per storage. -If you are using NFS to share Git data, we recommend that you enable a -number of feature flags that will allow GitLab application processes to -access Git data directly instead of going through the [Gitaly -service](../gitaly/index.md). Depending on your workload and disk -performance, these flags may help improve performance. See [the -issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/57317) for more -details. - -To do this, run the Rake task: +If you previously enabled Rugged using the feature flag, you will need to unset the feature flag by using: ```sh -sudo gitlab-rake gitlab:features:enable_rugged +sudo gitlab-rake gitlab:features:unset_rugged ``` -If you need to undo this setting for some reason such as switching to [Gitaly without NFS](gitaly.md) -(recommended), run: - -```sh -sudo gitlab-rake gitlab:features:disable_rugged -``` +If the Rugged feature flag is explicitly set to either true or false, GitLab will use the value explicitly set. ### Known issues @@ -81,16 +71,21 @@ bug](https://bugzilla.redhat.com/show_bug.cgi?id=1552203) that may be fixed in [more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). +NOTE: **Note** Red Hat Enterprise 7 [shipped a kernel +update](https://access.redhat.com/errata/RHSA-2019:2029) on August 6, +2019 that may have resolved this problem. The following instructions may +not be needed if the latest kernel is updated properly. + GitLab recommends all NFS users disable the NFS server delegation feature. To disable NFS server delegations on an Linux NFS server, do the following: 1. On the NFS server, run: - ```sh - echo 0 > /proc/sys/fs/leases-enable - sysctl -w fs.leases-enable=0 - ``` + ```sh + echo 0 > /proc/sys/fs/leases-enable + sysctl -w fs.leases-enable=0 + ``` 1. Restart the NFS server process. For example, on CentOS run `service nfs restart`. @@ -235,4 +230,16 @@ Read more on high-availability configuration: 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) -[udp-log-shipping]: http://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping" +[udp-log-shipping]: https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping" + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/nfs_host_client_setup.md b/doc/administration/high_availability/nfs_host_client_setup.md index a8d69b9ab0a..9b0e085fe25 100644 --- a/doc/administration/high_availability/nfs_host_client_setup.md +++ b/doc/administration/high_availability/nfs_host_client_setup.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Configuring NFS for GitLab HA Setting up NFS for a GitLab HA setup allows all applications nodes in a cluster @@ -133,3 +137,15 @@ client with the command below. ```sh sudo ufw allow from <client-ip-address> to any port nfs ``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 762179cf756..b99724d12a2 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -1,6 +1,8 @@ -# Working with the bundle Pgbouncer service +--- +type: reference +--- -## Overview +# Working with the bundle Pgbouncer service As part of its High Availability stack, GitLab Premium includes a bundled version of [Pgbouncer](https://pgbouncer.github.io/) that can be managed through `/etc/gitlab/gitlab.rb`. @@ -14,7 +16,89 @@ It is recommended to run pgbouncer alongside the `gitlab-rails` service, or on i ### Running Pgbouncer as part of an HA GitLab installation -See our [HA documentation for PostgreSQL](database.md) for information on running pgbouncer as part of a HA setup +1. Make sure you collect [`CONSUL_SERVER_NODES`](database.md#consul-information), [`CONSUL_PASSWORD_HASH`](database.md#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](database.md#pgbouncer-information) before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure Pgbouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # Replace CONSUL_PASSWORD_HASH with with a generated md5 value + # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value + pgbouncer['users'] = { + 'gitlab-consul': { + password: 'CONSUL_PASSWORD_HASH' + }, + 'pgbouncer': { + password: 'PGBOUNCER_PASSWORD_HASH' + } + } + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + NOTE: **Note:** + `pgbouncer_role` was introduced with GitLab 10.3. + +1. Run `gitlab-ctl reconfigure` + +1. Create a `.pgpass` file so Consul is able to + reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: + + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +#### PGBouncer Checkpoint + +1. Ensure the node is talking to the current master: + + ```sh + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + + If there is an error `psql: ERROR: Auth failed` after typing in the + password, ensure you previously generated the MD5 password hashes with the correct + format. The correct format is to concatenate the password and the username: + `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text + needed to generate an MD5 password hash for the `pgbouncer` user. + +1. Once the console prompt is available, run the following queries: + + ```sh + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ``` + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` ### Running Pgbouncer as part of a non-HA GitLab installation @@ -24,44 +108,71 @@ See our [HA documentation for PostgreSQL](database.md) for information on runnin 1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb` - ```ruby - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' - postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' - postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on - postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node - ``` + ```ruby + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' + postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' + postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on + postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node + ``` 1. Run `gitlab-ctl reconfigure` - **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. + **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. 1. On the node you are running pgbouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb` - ```ruby - pgbouncer['enable'] = true - pgbouncer['databases'] = { - gitlabhq_production: { - host: 'DATABASE_HOST', - user: 'pgbouncer', - password: 'PGBOUNCER_USER_PASSWORD_HASH' - } - } - ``` + ```ruby + pgbouncer['enable'] = true + pgbouncer['databases'] = { + gitlabhq_production: { + host: 'DATABASE_HOST', + user: 'pgbouncer', + password: 'PGBOUNCER_USER_PASSWORD_HASH' + } + } + ``` 1. Run `gitlab-ctl reconfigure` 1. On the node running unicorn, make sure the following is set in `/etc/gitlab/gitlab.rb` - ```ruby - gitlab_rails['db_host'] = 'PGBOUNCER_HOST' - gitlab_rails['db_port'] = '6432' - gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' - ``` + ```ruby + gitlab_rails['db_host'] = 'PGBOUNCER_HOST' + gitlab_rails['db_port'] = '6432' + gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' + ``` 1. Run `gitlab-ctl reconfigure` 1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** pgbouncer servers. + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ### Interacting with pgbouncer #### Administrative console @@ -82,6 +193,7 @@ pgbouncer=# The password you will be prompted for is the PGBOUNCER_USER_PASSWORD To get some basic information about the instance, run + ```shell pgbouncer=# show databases; show clients; show servers; name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 1aaa709fc8f..1b79dde9476 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,6 +1,10 @@ +--- +type: reference +--- + # Configuring Redis for Scaling and High Availability -## Provide your own Redis instance **[CORE ONLY]** +## Provide your own Redis instance **(CORE ONLY)** The following are the requirements for providing your own Redis instance: @@ -20,14 +24,14 @@ This section is relevant for [Scaled Architecture](README.md#scalable-architectu environments including [Basic Scaling](README.md#basic-scaling) and [Full Scaling](README.md#full-scaling). -### Provide your own Redis instance **[CORE ONLY]** +### Provide your own Redis instance **(CORE ONLY)** -If you want to use your own deployed Redis instance(s), -see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily -deploy the bundled Redis. +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. -### Standalone Redis using GitLab Omnibus **[CORE ONLY]** +### Standalone Redis using GitLab Omnibus **(CORE ONLY)** The GitLab Omnibus package can be used to configure a standalone Redis server. In this configuration Redis is not highly available, and represents a single @@ -41,39 +45,40 @@ The steps below are the minimum necessary to configure a Redis server with Omnibus: 1. SSH into the Redis server. -1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - ```ruby - ## Enable Redis - redis['enable'] = true - - ## Disable all other services - sidekiq['enable'] = false - gitlab_workhorse['enable'] = false - unicorn['enable'] = false - postgresql['enable'] = false - nginx['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - gitlab_monitor['enable'] = false - gitaly['enable'] = false - - redis['bind'] = '0.0.0.0' - redis['port'] = '6379' - redis['password'] = 'SECRET_PASSWORD_HERE' - - gitlab_rails['auto_migrate'] = false - ``` + ```ruby + ## Enable Redis + redis['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + gitlab_monitor['enable'] = false + gitaly['enable'] = false + + redis['bind'] = '0.0.0.0' + redis['port'] = '6379' + redis['password'] = 'SECRET_PASSWORD_HERE' + + gitlab_rails['auto_migrate'] = false + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and Redis password. These will be necessary when configuring the GitLab application servers later. +1. [Enable Monitoring](#enable-monitoring) Advanced configuration options are supported and can be added if needed. @@ -88,14 +93,14 @@ environments including [Horizontal](README.md#horizontal), [Hybrid](README.md#hybrid), and [Fully Distributed](README.md#fully-distributed). -### Provide your own Redis instance **[CORE ONLY]** +### Provide your own Redis instance **(CORE ONLY)** -If you want to use your own deployed Redis instance(s), -see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily -deploy the bundled Redis. +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. -### High Availability with GitLab Omnibus **[PREMIUM ONLY]** +### High Availability with GitLab Omnibus **(PREMIUM ONLY)** > Experimental Redis Sentinel support was [introduced in GitLab 8.11][ce-1877]. Starting with 8.14, Redis Sentinel is no longer experimental. @@ -356,95 +361,94 @@ The prerequisites for a HA Redis setup are the following: ### Step 1. Configuring the master Redis instance 1. SSH into the **master** Redis server. -1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] - - # IP address pointing to a local IP that the other machines can reach to. - # You can also set bind to '0.0.0.0' which listen in all interfaces. - # If you really need to bind to an external accessible IP, make - # sure you add extra firewall rules to prevent unauthorized access. - redis['bind'] = '10.0.0.1' - - # Define a port so Redis can listen for TCP requests which will allow other - # machines to connect to it. - redis['port'] = 6379 - - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'redis-password-goes-here' - ``` - + ```ruby + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.0.0.1' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'redis-password-goes-here' + ``` 1. Only the primary GitLab application server should handle migrations. To prevent database migrations from running on upgrade, add the following configuration to your `/etc/gitlab/gitlab.rb` file: - ``` - gitlab_rails['auto_migrate'] = false - ``` + ``` + gitlab_rails['auto_migrate'] = false + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -> Note: You can specify multiple roles like sentinel and redis as: -> roles ['redis_sentinel_role', 'redis_master_role']. Read more about high -> availability roles at https://docs.gitlab.com/omnibus/roles/ +> Note: You can specify multiple roles like sentinel and redis as: +> `roles ['redis_sentinel_role', 'redis_master_role']`. Read more about high +> availability roles at <https://docs.gitlab.com/omnibus/roles/>. ### Step 2. Configuring the slave Redis instances 1. SSH into the **slave** Redis server. -1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - ```ruby - # Specify server role as 'redis_slave_role' - roles ['redis_slave_role'] - - # IP address pointing to a local IP that the other machines can reach to. - # You can also set bind to '0.0.0.0' which listen in all interfaces. - # If you really need to bind to an external accessible IP, make - # sure you add extra firewall rules to prevent unauthorized access. - redis['bind'] = '10.0.0.2' + ```ruby + # Specify server role as 'redis_slave_role' + roles ['redis_slave_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.0.0.2' - # Define a port so Redis can listen for TCP requests which will allow other - # machines to connect to it. - redis['port'] = 6379 + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 - # The same password for Redis authentication you set up for the master node. - redis['password'] = 'redis-password-goes-here' + # The same password for Redis authentication you set up for the master node. + redis['password'] = 'redis-password-goes-here' - # The IP of the master Redis node. - redis['master_ip'] = '10.0.0.1' + # The IP of the master Redis node. + redis['master_ip'] = '10.0.0.1' - # Port of master Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ``` + # Port of master Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + ``` 1. To prevent reconfigure from running automatically on upgrade, run: - ``` - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` + ``` + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other slave nodes. -> Note: You can specify multiple roles like sentinel and redis as: -> roles ['redis_sentinel_role', 'redis_slave_role']. Read more about high -> availability roles at https://docs.gitlab.com/omnibus/roles/ +> Note: You can specify multiple roles like sentinel and redis as: +> `roles ['redis_sentinel_role', 'redis_slave_role']`. Read more about high +> availability roles at <https://docs.gitlab.com/omnibus/roles/>. --- @@ -487,89 +491,89 @@ multiple machines with the Sentinel daemon. 1. **You can omit this step if the Sentinels will be hosted in the same node as the other Redis instances.** - [Download/install](https://about.gitlab.com/downloads-ee) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. + [Download/install](https://about.gitlab.com/downloads-ee) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents (if you are installing the Sentinels in the same node as the other Redis instances, some values might be duplicate below): - ```ruby - roles ['redis_sentinel_role'] - - # Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis' - - # The same password for Redis authentication you set up for the master node. - redis['master_password'] = 'redis-password-goes-here' - - # The IP of the master Redis node. - redis['master_ip'] = '10.0.0.1' - - # Define a port so Redis can listen for TCP requests which will allow other - # machines to connect to it. - redis['port'] = 6379 - - # Port of master Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel - sentinel['bind'] = '10.0.0.1' - - # Port that Sentinel listens on, uncomment to change to non default. Defaults - # to `26379`. - # sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to master failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the master. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the master being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - # sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same master by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a slave replicating to a wrong master according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right master, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (SLAVEOF NO ONE yet not - ## acknowledged by the promoted slave). - ## - ## - The maximum time a failover in progress waits for all the slaves to be - ## reconfigured as slaves of the new master. However even after this time - ## the slaves will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - # sentinel['failover_timeout'] = 60000 - ``` + ```ruby + roles ['redis_sentinel_role'] + + # Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' + + # The same password for Redis authentication you set up for the master node. + redis['master_password'] = 'redis-password-goes-here' + + # The IP of the master Redis node. + redis['master_ip'] = '10.0.0.1' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Port of master Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel + sentinel['bind'] = '10.0.0.1' + + # Port that Sentinel listens on, uncomment to change to non default. Defaults + # to `26379`. + # sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to master failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the master. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the master being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + # sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same master by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a slave replicating to a wrong master according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right master, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (SLAVEOF NO ONE yet not + ## acknowledged by the promoted slave). + ## + ## - The maximum time a failover in progress waits for all the slaves to be + ## reconfigured as slaves of the new master. However even after this time + ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + # sentinel['failover_timeout'] = 60000 + ``` 1. To prevent database migrations from running on upgrade, run: - ``` - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` + ``` + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` - Only the primary GitLab application server should handle migrations. + Only the primary GitLab application server should handle migrations. 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other Sentinel nodes. @@ -593,20 +597,20 @@ which ideally should not have Redis or Sentinels on it for a HA setup. 1. SSH into the server where the GitLab application is installed. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: - ``` - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis' + ```ruby + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' - ## The same password for Redis authentication you set up for the master node. - redis['master_password'] = 'redis-password-goes-here' + ## The same password for Redis authentication you set up for the master node. + redis['master_password'] = 'redis-password-goes-here' - ## A list of sentinels with `host` and `port` - gitlab_rails['redis_sentinels'] = [ - {'host' => '10.0.0.1', 'port' => 26379}, - {'host' => '10.0.0.2', 'port' => 26379}, - {'host' => '10.0.0.3', 'port' => 26379} - ] - ``` + ## A list of sentinels with `host` and `port` + gitlab_rails['redis_sentinels'] = [ + {'host' => '10.0.0.1', 'port' => 26379}, + {'host' => '10.0.0.2', 'port' => 26379}, + {'host' => '10.0.0.3', 'port' => 26379} + ] + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. @@ -749,6 +753,35 @@ gitlab_rails['redis_sentinels'] = [ [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** Redis servers. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ## Advanced configuration Omnibus GitLab configures some things behind the curtains to make the sysadmins' @@ -762,31 +795,34 @@ cache, queues, and shared_state. To make this work with Sentinel: 1. Set the appropriate variable in `/etc/gitlab/gitlab.rb` for each instance you are using: - ```ruby - gitlab_rails['redis_cache_instance'] = REDIS_CACHE_URL - gitlab_rails['redis_queues_instance'] = REDIS_QUEUES_URL - gitlab_rails['redis_shared_state_instance'] = REDIS_SHARED_STATE_URL - ``` + ```ruby + gitlab_rails['redis_cache_instance'] = REDIS_CACHE_URL + gitlab_rails['redis_queues_instance'] = REDIS_QUEUES_URL + gitlab_rails['redis_shared_state_instance'] = REDIS_SHARED_STATE_URL + ``` + **Note**: Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_MASTER_NAME` - 1. PASSWORD is the plaintext password for the Redis instance - 1. SENTINEL_MASTER_NAME is the Sentinel master name (e.g. `gitlab-redis-cache`) + 1. PASSWORD is the plaintext password for the Redis instance + 1. SENTINEL_MASTER_NAME is the Sentinel master name (e.g. `gitlab-redis-cache`) + 1. Include an array of hashes with host/port combinations, such as the following: - ```ruby - gitlab_rails['redis_cache_sentinels'] = [ - { host: REDIS_CACHE_SENTINEL_HOST, port: PORT1 }, - { host: REDIS_CACHE_SENTINEL_HOST2, port: PORT2 } - ] - gitlab_rails['redis_queues_sentinels'] = [ - { host: REDIS_QUEUES_SENTINEL_HOST, port: PORT1 }, - { host: REDIS_QUEUES_SENTINEL_HOST2, port: PORT2 } - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - { host: SHARED_STATE_SENTINEL_HOST, port: PORT1 }, - { host: SHARED_STATE_SENTINEL_HOST2, port: PORT2 } - ] - ``` + ```ruby + gitlab_rails['redis_cache_sentinels'] = [ + { host: REDIS_CACHE_SENTINEL_HOST, port: PORT1 }, + { host: REDIS_CACHE_SENTINEL_HOST2, port: PORT2 } + ] + gitlab_rails['redis_queues_sentinels'] = [ + { host: REDIS_QUEUES_SENTINEL_HOST, port: PORT1 }, + { host: REDIS_QUEUES_SENTINEL_HOST2, port: PORT2 } + ] + gitlab_rails['redis_shared_state_sentinels'] = [ + { host: SHARED_STATE_SENTINEL_HOST, port: PORT1 }, + { host: SHARED_STATE_SENTINEL_HOST2, port: PORT2 } + ] + ``` + 1. Note that for each persistence class, GitLab will default to using the configuration specified in `gitlab_rails['redis_sentinels']` unless overridden by the settings above. @@ -850,12 +886,12 @@ in order for the HA setup to work as expected. Before proceeding with the troubleshooting below, check your firewall rules: - Redis machines - - Accept TCP connection in `6379` - - Connect to the other Redis machines via TCP in `6379` + - Accept TCP connection in `6379` + - Connect to the other Redis machines via TCP in `6379` - Sentinel machines - - Accept TCP connection in `26379` - - Connect to other Sentinel machines via TCP in `26379` - - Connect to the Redis machines via TCP in `6379` + - Accept TCP connection in `26379` + - Connect to other Sentinel machines via TCP in `26379` + - Connect to the Redis machines via TCP in `6379` ### Troubleshooting Redis replication @@ -923,38 +959,38 @@ To make sure your configuration is correct: 1. SSH into your GitLab application server 1. Enter the Rails console: - ``` - # For Omnibus installations - sudo gitlab-rails console + ``` + # For Omnibus installations + sudo gitlab-rails console - # For source installations - sudo -u git rails console production - ``` + # For source installations + sudo -u git rails console production + ``` 1. Run in the console: - ```ruby - redis = Redis.new(Gitlab::Redis::SharedState.params) - redis.info - ``` + ```ruby + redis = Redis.new(Gitlab::Redis::SharedState.params) + redis.info + ``` - Keep this screen open and try to simulate a failover below. + Keep this screen open and try to simulate a failover below. 1. To simulate a failover on master Redis, SSH into the Redis server and run: - ```bash - # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one - redis-cli -h localhost -p 6379 DEBUG sleep 20 - ``` + ```bash + # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one + redis-cli -h localhost -p 6379 DEBUG sleep 20 + ``` 1. Then back in the Rails console from the first step, run: - ``` - redis.info - ``` + ``` + redis.info + ``` - You should see a different port after a few seconds delay - (the failover/reconnect time). + You should see a different port after a few seconds delay + (the failover/reconnect time). ## Changelog diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index be6b547372a..63915e5d96c 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Configuring non-Omnibus Redis for GitLab HA This is the documentation for configuring a Highly Available Redis setup when @@ -49,22 +53,22 @@ Assuming that the Redis master instance IP is `10.0.0.1`: 1. [Install Redis](../../install/installation.md#7-redis). 1. Edit `/etc/redis/redis.conf`: - ```conf - ## Define a `bind` address pointing to a local IP that your other machines - ## can reach you. If you really need to bind to an external accessible IP, make - ## sure you add extra firewall rules to prevent unauthorized access: - bind 10.0.0.1 + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.1 - ## Define a `port` to force redis to listen on TCP so other machines can - ## connect to it (default port is `6379`). - port 6379 + ## Define a `port` to force redis to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 6379 - ## Set up password authentication (use the same password in all nodes). - ## The password should be defined equal for both `requirepass` and `masterauth` - ## when setting up Redis to use with Sentinel. - requirepass redis-password-goes-here - masterauth redis-password-goes-here - ``` + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + ``` 1. Restart the Redis service for the changes to take effect. @@ -75,25 +79,25 @@ Assuming that the Redis slave instance IP is `10.0.0.2`: 1. [Install Redis](../../install/installation.md#7-redis). 1. Edit `/etc/redis/redis.conf`: - ```conf - ## Define a `bind` address pointing to a local IP that your other machines - ## can reach you. If you really need to bind to an external accessible IP, make - ## sure you add extra firewall rules to prevent unauthorized access: - bind 10.0.0.2 + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.2 - ## Define a `port` to force redis to listen on TCP so other machines can - ## connect to it (default port is `6379`). - port 6379 + ## Define a `port` to force redis to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 6379 - ## Set up password authentication (use the same password in all nodes). - ## The password should be defined equal for both `requirepass` and `masterauth` - ## when setting up Redis to use with Sentinel. - requirepass redis-password-goes-here - masterauth redis-password-goes-here + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here - ## Define `slaveof` pointing to the Redis master instance with IP and port. - slaveof 10.0.0.1 6379 - ``` + ## Define `slaveof` pointing to the Redis master instance with IP and port. + slaveof 10.0.0.1 6379 + ``` 1. Restart the Redis service for the changes to take effect. 1. Go through the steps again for all the other slave nodes. @@ -110,56 +114,57 @@ master with IP `10.0.0.1` (some settings might overlap with the master): 1. [Install Redis Sentinel](https://redis.io/topics/sentinel) 1. Edit `/etc/redis/sentinel.conf`: - ```conf - ## Define a `bind` address pointing to a local IP that your other machines - ## can reach you. If you really need to bind to an external accessible IP, make - ## sure you add extra firewall rules to prevent unauthorized access: - bind 10.0.0.1 - - ## Define a `port` to force Sentinel to listen on TCP so other machines can - ## connect to it (default port is `6379`). - port 26379 - - ## Set up password authentication (use the same password in all nodes). - ## The password should be defined equal for both `requirepass` and `masterauth` - ## when setting up Redis to use with Sentinel. - requirepass redis-password-goes-here - masterauth redis-password-goes-here - - ## Define with `sentinel auth-pass` the same shared password you have - ## defined for both Redis master and slaves instances. - sentinel auth-pass gitlab-redis redis-password-goes-here - - ## Define with `sentinel monitor` the IP and port of the Redis - ## master node, and the quorum required to start a failover. - sentinel monitor gitlab-redis 10.0.0.1 6379 2 - - ## Define with `sentinel down-after-milliseconds` the time in `ms` - ## that an unresponsive server will be considered down. - sentinel down-after-milliseconds gitlab-redis 10000 - - ## Define a value for `sentinel failover_timeout` in `ms`. This has multiple - ## meanings: - ## - ## * The time needed to re-start a failover after a previous failover was - ## already tried against the same master by a given Sentinel, is two - ## times the failover timeout. - ## - ## * The time needed for a slave replicating to a wrong master according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right master, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## * The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (SLAVEOF NO ONE yet not - ## acknowledged by the promoted slave). - ## - ## * The maximum time a failover in progress waits for all the slaves to be - ## reconfigured as slaves of the new master. However even after this time - ## the slaves will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - sentinel failover_timeout 30000 - ``` + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.1 + + ## Define a `port` to force Sentinel to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 26379 + + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + + ## Define with `sentinel auth-pass` the same shared password you have + ## defined for both Redis master and slaves instances. + sentinel auth-pass gitlab-redis redis-password-goes-here + + ## Define with `sentinel monitor` the IP and port of the Redis + ## master node, and the quorum required to start a failover. + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + + ## Define with `sentinel down-after-milliseconds` the time in `ms` + ## that an unresponsive server will be considered down. + sentinel down-after-milliseconds gitlab-redis 10000 + + ## Define a value for `sentinel failover_timeout` in `ms`. This has multiple + ## meanings: + ## + ## * The time needed to re-start a failover after a previous failover was + ## already tried against the same master by a given Sentinel, is two + ## times the failover timeout. + ## + ## * The time needed for a slave replicating to a wrong master according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right master, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## * The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (SLAVEOF NO ONE yet not + ## acknowledged by the promoted slave). + ## + ## * The maximum time a failover in progress waits for all the slaves to be + ## reconfigured as slaves of the new master. However even after this time + ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + sentinel failover_timeout 30000 + ``` + 1. Restart the Redis service for the changes to take effect. 1. Go through the steps again for all the other Sentinel nodes. @@ -180,21 +185,21 @@ setup: [resque.yml.example][resque], and uncomment the Sentinel lines, pointing to the correct server credentials: - ```yaml - # resque.yaml - production: - url: redis://:redi-password-goes-here@gitlab-redis/ - sentinels: - - - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port - - - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port - - - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port - ``` + ```yaml + # resque.yaml + production: + url: redis://:redi-password-goes-here@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port + ``` 1. [Restart GitLab][restart] for the changes to take effect. @@ -232,23 +237,23 @@ or a failover promotes a different **Master** node. 1. In `/etc/redis/redis.conf`: - ```conf - bind 10.0.0.1 - port 6379 - requirepass redis-password-goes-here - masterauth redis-password-goes-here - ``` + ```conf + bind 10.0.0.1 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + ``` 1. In `/etc/redis/sentinel.conf`: - ```conf - bind 10.0.0.1 - port 26379 - sentinel auth-pass gitlab-redis redis-password-goes-here - sentinel monitor gitlab-redis 10.0.0.1 6379 2 - sentinel down-after-milliseconds gitlab-redis 10000 - sentinel failover_timeout 30000 - ``` + ```conf + bind 10.0.0.1 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` 1. Restart the Redis service for the changes to take effect. @@ -256,24 +261,24 @@ or a failover promotes a different **Master** node. 1. In `/etc/redis/redis.conf`: - ```conf - bind 10.0.0.2 - port 6379 - requirepass redis-password-goes-here - masterauth redis-password-goes-here - slaveof 10.0.0.1 6379 - ``` + ```conf + bind 10.0.0.2 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + slaveof 10.0.0.1 6379 + ``` 1. In `/etc/redis/sentinel.conf`: - ```conf - bind 10.0.0.2 - port 26379 - sentinel auth-pass gitlab-redis redis-password-goes-here - sentinel monitor gitlab-redis 10.0.0.1 6379 2 - sentinel down-after-milliseconds gitlab-redis 10000 - sentinel failover_timeout 30000 - ``` + ```conf + bind 10.0.0.2 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` 1. Restart the Redis service for the changes to take effect. @@ -281,24 +286,24 @@ or a failover promotes a different **Master** node. 1. In `/etc/redis/redis.conf`: - ```conf - bind 10.0.0.3 - port 6379 - requirepass redis-password-goes-here - masterauth redis-password-goes-here - slaveof 10.0.0.1 6379 - ``` + ```conf + bind 10.0.0.3 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + slaveof 10.0.0.1 6379 + ``` 1. In `/etc/redis/sentinel.conf`: - ```conf - bind 10.0.0.3 - port 26379 - sentinel auth-pass gitlab-redis redis-password-goes-here - sentinel monitor gitlab-redis 10.0.0.1 6379 2 - sentinel down-after-milliseconds gitlab-redis 10000 - sentinel failover_timeout 30000 - ``` + ```conf + bind 10.0.0.3 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` 1. Restart the Redis service for the changes to take effect. @@ -306,20 +311,20 @@ or a failover promotes a different **Master** node. 1. Edit `/home/git/gitlab/config/resque.yml`: - ```yaml - production: - url: redis://:redi-password-goes-here@gitlab-redis/ - sentinels: - - - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port - - - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port - - - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port - ``` + ```yaml + production: + url: redis://:redi-password-goes-here@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port + ``` 1. [Restart GitLab][restart] for the changes to take effect. diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 1b01419e062..abe39d188aa 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -2,7 +2,6 @@ > [Introduced][ce-2371] in GitLab 8.4. ---- ## Automatic housekeeping GitLab automatically runs `git gc` and `git repack` on repositories @@ -32,8 +31,6 @@ the `pushes_since_gc` value is 200 a `git gc` will be run. You can find this option under your project's **Settings > General > Advanced**. ---- -  [ce-2371]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2371 "Housekeeping merge request" diff --git a/doc/administration/audit_log.png b/doc/administration/img/audit_log.png Binary files differindex d4f4c2abf38..d4f4c2abf38 100644 --- a/doc/administration/audit_log.png +++ b/doc/administration/img/audit_log.png diff --git a/doc/administration/auditor_access_form.png b/doc/administration/img/auditor_access_form.png Binary files differindex c179a7d3b0a..c179a7d3b0a 100644 --- a/doc/administration/auditor_access_form.png +++ b/doc/administration/img/auditor_access_form.png diff --git a/doc/administration/img/custom_hooks_error_msg.png b/doc/administration/img/custom_hooks_error_msg.png Binary files differindex 4f25c471908..e7d5c157d08 100644 --- a/doc/administration/img/custom_hooks_error_msg.png +++ b/doc/administration/img/custom_hooks_error_msg.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 8271c579f5b..73a39a6dd35 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -4,14 +4,14 @@ GitLab has several features based on receiving incoming emails: - [Reply by Email](reply_by_email.md): allow GitLab users to comment on issues and merge requests by replying to notification emails. -- [New issue by email](../user/project/issues/create_new_issue.md#new-issue-via-email): +- [New issue by email](../user/project/issues/managing_issues.md#new-issue-via-email): allow GitLab users to create a new issue by sending an email to a user-specific email address. - [New merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email): allow GitLab users to create a new merge request by sending an email to a user-specific email address. - [Service Desk](../user/project/service_desk.md): provide e-mail support to - your customers through GitLab. **[PREMIUM]** + your customers through GitLab. **(PREMIUM)** ## Requirements @@ -102,16 +102,16 @@ for a real-world example of this exploit. 1. Reconfigure GitLab for the changes to take effect: - ```sh - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` 1. Verify that everything is configured correctly: - ```sh - sudo gitlab-rake gitlab:incoming_email:check - ``` + ```sh + sudo gitlab-rake gitlab:incoming_email:check + ``` Reply by email should now be working. @@ -119,31 +119,31 @@ Reply by email should now be working. 1. Go to the GitLab installation directory: - ```sh - cd /home/git/gitlab - ``` + ```sh + cd /home/git/gitlab + ``` 1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and fill in the details for your specific IMAP server and email account (see [examples](#config-examples) below). 1. Enable `mail_room` in the init script at `/etc/default/gitlab`: - ```sh - sudo mkdir -p /etc/default - echo 'mail_room_enabled=true' | sudo tee -a /etc/default/gitlab - ``` + ```sh + sudo mkdir -p /etc/default + echo 'mail_room_enabled=true' | sudo tee -a /etc/default/gitlab + ``` 1. Restart GitLab: - ```sh - sudo service gitlab restart - ``` + ```sh + sudo service gitlab restart + ``` 1. Verify that everything is configured correctly: - ```sh - sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production - ``` + ```sh + sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production + ``` Reply by email should now be working. diff --git a/doc/administration/index.md b/doc/administration/index.md index 06d900b152d..2b25e8efd23 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -2,7 +2,7 @@ description: 'Learn how to install, configure, update, and maintain your GitLab instance.' --- -# Administrator documentation **[CORE ONLY]** +# Administrator Docs **(CORE ONLY)** Learn how to administer your self-managed GitLab instance. @@ -11,7 +11,7 @@ GitLab has two product distributions available through [different subscriptions] - The open source [GitLab Community Edition (CE)](https://gitlab.com/gitlab-org/gitlab-ce). - The open core [GitLab Enterprise Edition (EE)](https://gitlab.com/gitlab-org/gitlab-ee). -You can [install either GitLab CE or GitLab EE](https://about.gitlab.com/installation/ce-or-ee/). +You can [install either GitLab CE or GitLab EE](https://about.gitlab.com/install/ce-or-ee/). However, the features you'll have access to depend on the subscription you choose (Core, Starter, Premium, or Ultimate). @@ -32,15 +32,14 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Installing GitLab - [Install](../install/README.md): Requirements, directory structures, and installation methods. - - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. **[STARTER ONLY]** - - [Omnibus support for external MySQL DB](https://docs.gitlab.com/omnibus/settings/database.html#using-a-mysql-database-management-server-enterprise-edition-only): Omnibus package supports configuring an external MySQL database. **[STARTER ONLY]** - - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **[STARTER ONLY]** + - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. **(STARTER ONLY)** + - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **(STARTER ONLY)** - [High Availability](high_availability/README.md): Configure multiple servers for scaling or high availability. - [Installing GitLab HA on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab High Availability on Amazon AWS. -- [Geo](geo/replication/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **[PREMIUM ONLY]** -- [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. **[PREMIUM ONLY]** -- [Pivotal Tile](../install/pivotal/index.md): Deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **[PREMIUM ONLY]** -- [Add License](../user/admin_area/license.md): Upload a license at install time to unlock features that are in paid tiers of GitLab. **[STARTER ONLY]** +- [Geo](geo/replication/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **(PREMIUM ONLY)** +- [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. **(PREMIUM ONLY)** +- [Pivotal Tile](../install/pivotal/index.md): Deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **(PREMIUM ONLY)** +- [Add License](../user/admin_area/license.md): Upload a license at install time to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** ### Configuring GitLab @@ -61,9 +60,9 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering size limits of branch comparison pages. - [Merge request diffs storage](merge_request_diffs.md): Configure merge requests diffs external storage. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. -- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower GitLab's Advanced Global Search. Useful when you deal with a huge amount of data. **[STARTER ONLY]** -- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **[PREMIUM ONLY]** -- [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **[STARTER ONLY]** +- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower GitLab's Advanced Global Search. Useful when you deal with a huge amount of data. **(STARTER ONLY)** +- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **(PREMIUM ONLY)** +- [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. #### Customizing GitLab's appearance @@ -73,7 +72,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Branded login page](../customization/branded_login_page.md): Customize the login page with your own logo, title, and description. - [Welcome message](../customization/welcome_message.md): Add a custom welcome message to the sign-in page. - ["New Project" page](../customization/new_project_page.md): Customize the text to be displayed on the page that opens whenever your users create a new project. -- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text-premium-only): Add additional custom text to emails sent from GitLab. **[PREMIUM ONLY]** +- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text-premium-only): Add additional custom text to emails sent from GitLab. **(PREMIUM ONLY)** ### Maintaining GitLab @@ -108,17 +107,17 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS and additional providers. - - [Sync LDAP](auth/ldap-ee.md) **[STARTER ONLY]** - - [Kerberos authentication](../integration/kerberos.md) **[STARTER ONLY]** + - [Sync LDAP](auth/ldap-ee.md) **(STARTER ONLY)** + - [Kerberos authentication](../integration/kerberos.md) **(STARTER ONLY)** - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). -- [Email users](../tools/email.md): Email GitLab users from within GitLab. **[STARTER ONLY]** +- [Email users](../tools/email.md): Email GitLab users from within GitLab. **(STARTER ONLY)** - [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. - [Audit logs and events](audit_events.md): View the changes made within the GitLab server for: - - Groups and projects. **[STARTER]** - - Instances. **[PREMIUM ONLY]** -- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **[PREMIUM ONLY]** + - Groups and projects. **(STARTER)** + - Instances. **(PREMIUM ONLY)** +- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **(PREMIUM ONLY)** - [Incoming email](incoming_email.md): Configure incoming emails to allow - users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/create_new_issue.md#new-issue-via-email) and + users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/managing_issues.md#new-issue-via-email) and [merge requests by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email), and to enable [Service Desk](../user/project/service_desk.md). - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail server with IMAP authentication on Ubuntu for incoming @@ -132,15 +131,15 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Gitaly](gitaly/index.md): Configuring Gitaly, GitLab's Git repository storage service. - [Default labels](../user/admin_area/labels.md): Create labels that will be automatically added to every new project. - [Restrict the use of public or internal projects](../public_access/public_access.md#restricting-the-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. -- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. **[PREMIUM ONLY]** -- [Packages](packages.md): Enable GitLab to act as a Maven repository or NPM registry. **[PREMIUM ONLY]** +- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. **(PREMIUM ONLY)** +- [Packages](packages.md): Enable GitLab to act as a Maven repository or NPM registry. **(PREMIUM ONLY)** ### Repository settings - [Repository checks](repository_checks.md): Periodic Git repository checks. - [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. - [Repository storage rake tasks](raketasks/storage.md): A collection of rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. -- [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size. **[STARTER ONLY]** +- [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size. **(STARTER ONLY)** ## Continuous Integration settings @@ -149,7 +148,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job traces](job_traces.md): Information about the job traces (logs). - [Register Shared and specific Runners](../ci/runners/README.md#registering-a-shared-runner): Learn how to register and configure Shared and specific Runners to your own instance. -- [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota-starter-only): Limit the usage of pipeline minutes for Shared Runners. **[STARTER ONLY]** +- [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota-starter-only): Limit the usage of pipeline minutes for Shared Runners. **(STARTER ONLY)** - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enablingdisabling-auto-devops): Enable or disable Auto DevOps for your instance. ## Git configuration options @@ -179,10 +178,20 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Analytics -- [Pseudonymizer](pseudonymizer.md): Export data from GitLab's database to CSV files in a secure way. **[ULTIMATE]** +- [Pseudonymizer](pseudonymizer.md): Export data from GitLab's database to CSV files in a secure way. **(ULTIMATE)** ## Troubleshooting - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. +- Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab + Support team. +- [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md): Tips to troubleshoot ElasticSearch. +- [Kubernetes troubleshooting](troubleshooting/kubernetes_cheat_sheet.md): Commands and tips useful + for troubleshooting Kubernetes-related issues. +- Useful links from the Support Team: + - [GitLab Developer Docs](https://docs.gitlab.com/ee/development/README.html). + - [Repairing and recovering broken git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html). + - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html). + - [Strace zine](https://wizardzines.com/zines/strace/). diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index b1244f44e95..ab6a4646a71 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -1,4 +1,4 @@ -# Instance Review **[CORE ONLY]** +# Instance Review **(CORE ONLY)** > [Introduced][6995] in [GitLab Core][ee] 11.3. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 82e0c14ffc2..16a193550a1 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -1,6 +1,7 @@ # PlantUML & GitLab -> [Introduced][ce-8537] in GitLab 8.16. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8537) in +> GitLab 8.16. When [PlantUML](http://plantuml.com) integration is enabled and configured in GitLab we are able to create simple diagrams in AsciiDoc and Markdown documents @@ -15,7 +16,9 @@ server that will generate the diagrams. With Docker, you can just run a container like this: -`docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat` +```sh +docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat +``` The **PlantUML URL** will be the hostname of the server running the container. @@ -26,7 +29,7 @@ own PlantUML server is easy in Debian/Ubuntu distributions using Tomcat. First you need to create a `plantuml.war` file from the source code: -``` +```sh sudo apt-get install graphviz openjdk-8-jdk git-core maven git clone https://github.com/plantuml/plantuml-server.git cd plantuml-server @@ -36,7 +39,7 @@ mvn package The above sequence of commands will generate a WAR file that can be deployed using Tomcat: -``` +```sh sudo apt-get install tomcat7 sudo cp target/plantuml.war /var/lib/tomcat7/webapps/plantuml.war sudo chown tomcat7:tomcat7 /var/lib/tomcat7/webapps/plantuml.war @@ -46,7 +49,7 @@ sudo service tomcat7 restart Once the Tomcat service restarts the PlantUML service will be ready and listening for requests on port 8080: -``` +```text http://localhost:8080/plantuml ``` @@ -57,9 +60,10 @@ you can change these defaults by editing the `/etc/tomcat7/server.xml` file. You need to enable PlantUML integration from Settings under Admin Area. To do that, login with an Admin account and do following: - - in GitLab go to **Admin Area**->**Settings**->**Integrations**->**PlantUML** - - check **Enable PlantUML** checkbox - - set the PlantUML instance as **PlantUML URL** +- In GitLab, go to **Admin Area > Settings > Integrations**. +- Expand the **PlantUML** section. +- Check **Enable PlantUML** checkbox. +- Set the PlantUML instance as **PlantUML URL**. ## Creating Diagrams @@ -68,33 +72,34 @@ our AsciiDoc snippets, wikis and repos using delimited blocks: - **Markdown** - <pre> - ```plantuml - Bob -> Alice : hello - Alice -> Bob : Go Away - ```</pre> + ~~~markdown + ```plantuml + Bob -> Alice : hello + Alice -> Bob : Go Away + ``` + ~~~ - **AsciiDoc** - ``` - [plantuml, format="png", id="myDiagram", width="200px"] - ---- - Bob->Alice : hello - Alice -> Bob : Go Away - ---- - ``` + ```text + [plantuml, format="png", id="myDiagram", width="200px"] + ---- + Bob->Alice : hello + Alice -> Bob : Go Away + ---- + ``` - **reStructuredText** - ``` - .. plantuml:: - :caption: Caption with **bold** and *italic* + ```text + .. plantuml:: + :caption: Caption with **bold** and *italic* - Bob -> Alice: hello - Alice -> Bob: Go Away - ``` + Bob -> Alice: hello + Alice -> Bob: Go Away + ``` - You can also use the `uml::` directive for compatibility with [sphinxcontrib-plantuml](https://pypi.python.org/pypi/sphinxcontrib-plantuml), but please note that we currently only support the `caption` option. + You can also use the `uml::` directive for compatibility with [sphinxcontrib-plantuml](https://pypi.org/project/sphinxcontrib-plantuml/), but please note that we currently only support the `caption` option. The above blocks will be converted to an HTML img tag with source pointing to the PlantUML instance. If the PlantUML server is correctly configured, this should @@ -111,12 +116,10 @@ diagram delimiters `@startuml`/`@enduml` as these are replaced by the AsciiDoc ` Some parameters can be added to the AsciiDoc block definition: - - *format*: Can be either `png` or `svg`. Note that `svg` is not supported by - all browsers so use with care. The default is `png`. - - *id*: A CSS id added to the diagram HTML tag. - - *width*: Width attribute added to the img tag. - - *height*: Height attribute added to the img tag. +- *format*: Can be either `png` or `svg`. Note that `svg` is not supported by + all browsers so use with care. The default is `png`. +- *id*: A CSS id added to the diagram HTML tag. +- *width*: Width attribute added to the img tag. +- *height*: Height attribute added to the img tag. Markdown does not support any parameters and will always use PNG format. - -[ce-8537]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8537 diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index c34858cd0db..24c9cc0bea9 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -41,9 +41,9 @@ detail below. - Every session URL that is created has an authorization header that needs to be sent, to establish a `wss` connection. - The session URL is not exposed to the users in any way. GitLab holds all the state internally and proxies accordingly. -## Enabling and disabling terminal support +## Enabling and disabling terminal support -NOTE: **Note:** AWS Elastic Load Balancers (ELBs) do not support web sockets. +NOTE: **Note:** AWS Elastic Load Balancers (ELBs) do not support web sockets. AWS Application Load Balancers (ALBs) must be used if you want web terminals to work. See [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare) for more information. diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 160da47c780..e1bbabb2878 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -1,4 +1,4 @@ -# Issue closing pattern +# Issue closing pattern **(CORE ONLY)** >**Note:** This is the administration documentation. @@ -27,9 +27,10 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by 1. Change the value of `gitlab_rails['gitlab_issue_closing_pattern']` to a regular expression of your liking: - ```ruby - gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" - ``` + ```ruby + gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" + ``` + 1. [Reconfigure] GitLab for the changes to take effect. **For installations from source** @@ -37,13 +38,13 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by 1. Open `gitlab.yml` with your editor. 1. Change the value of `issue_closing_pattern`: - ```yaml - issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" - ``` + ```yaml + issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" + ``` 1. [Restart] GitLab for the changes to take effect. [gitlab.yml.example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example [reconfigure]: restart_gitlab.md#omnibus-gitlab-reconfigure [restart]: restart_gitlab.md#installations-from-source -[user documentation]: ../user/project/issues/automatic_issue_closing.md +[user documentation]: ../user/project/issues/managing_issues.md#closing-issues-automatically diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 05e15fc303b..2b624f8de77 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -1,7 +1,5 @@ # Jobs artifacts administration -> **Notes:** -> > - Introduced in GitLab 8.2 and GitLab Runner 0.7.0. > - Starting with GitLab 8.4 and GitLab Runner 1.0, the artifacts archive format changed to `ZIP`. > - Starting with GitLab 8.17, builds are renamed to jobs. @@ -15,28 +13,24 @@ if you want to know how to disable it. To disable artifacts site-wide, follow the steps below. ---- - **In Omnibus installations:** 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['artifacts_enabled'] = false - ``` + ```ruby + gitlab_rails['artifacts_enabled'] = false + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **In installations from source:** 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - artifacts: - enabled: false - ``` + ```yaml + artifacts: + enabled: false + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -51,8 +45,6 @@ this is done when the job succeeds, but can also be done on failure, or always, To change the location where the artifacts are stored locally, follow the steps below. ---- - **In Omnibus installations:** _The artifacts are stored by default in @@ -61,14 +53,12 @@ _The artifacts are stored by default in 1. To change the storage path for example to `/mnt/storage/artifacts`, edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['artifacts_path'] = "/mnt/storage/artifacts" - ``` + ```ruby + gitlab_rails['artifacts_path'] = "/mnt/storage/artifacts" + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **In installations from source:** _The artifacts are stored by default in @@ -77,18 +67,16 @@ _The artifacts are stored by default in 1. To change the storage path for example to `/mnt/storage/artifacts`, edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - artifacts: - enabled: true - path: /mnt/storage/artifacts - ``` + ```yaml + artifacts: + enabled: true + path: /mnt/storage/artifacts + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. ### Using object storage -> **Notes:** -> > - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1762) in > [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. > - Since version 9.5, artifacts are [browsable](../user/project/pipelines/job_artifacts.md#browsing-artifacts), @@ -127,6 +115,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -141,37 +130,35 @@ _The artifacts are stored by default in 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: - ```ruby - gitlab_rails['artifacts_enabled'] = true - gitlab_rails['artifacts_object_store_enabled'] = true - gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts" - gitlab_rails['artifacts_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', - 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' - } - ``` - - NOTE: For GitLab 9.4+, if you are using AWS IAM profiles, be sure to omit the - AWS access key and secret access key/value pairs. For example: - - ```ruby - gitlab_rails['artifacts_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'use_iam_profile' => true - } - ``` + ```ruby + gitlab_rails['artifacts_enabled'] = true + gitlab_rails['artifacts_object_store_enabled'] = true + gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts" + gitlab_rails['artifacts_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: For GitLab 9.4+, if you are using AWS IAM profiles, be sure to omit the + AWS access key and secret access key/value pairs. For example: + + ```ruby + gitlab_rails['artifacts_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. 1. Migrate any existing local artifacts to the object storage: - ```bash - gitlab-rake gitlab:artifacts:migrate - ``` - ---- + ```bash + gitlab-rake gitlab:artifacts:migrate + ``` **In installations from source:** @@ -181,25 +168,25 @@ _The artifacts are stored by default in 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - artifacts: - enabled: true - object_store: - enabled: true - remote_directory: "artifacts" # The bucket name - connection: - provider: AWS # Only AWS supported at the moment - aws_access_key_id: AWS_ACCESS_KEY_ID - aws_secret_access_key: AWS_SECRET_ACCESS_KEY - region: eu-central-1 - ``` + ```yaml + artifacts: + enabled: true + object_store: + enabled: true + remote_directory: "artifacts" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Migrate any existing local artifacts to the object storage: - ```bash - sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production - ``` + ```bash + sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production + ``` ## Expiring artifacts @@ -211,29 +198,25 @@ right after that date passes. Artifacts are cleaned up by the To change the default schedule on which the artifacts are expired, follow the steps below. ---- - **In Omnibus installations:** 1. Edit `/etc/gitlab/gitlab.rb` and comment out or add the following line - ```ruby - gitlab_rails['expire_build_artifacts_worker_cron'] = "50 * * * *" - ``` + ```ruby + gitlab_rails['expire_build_artifacts_worker_cron'] = "50 * * * *" + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **In installations from source:** 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - expire_build_artifacts_worker: - cron: "50 * * * *" - ``` + ```yaml + expire_build_artifacts_worker: + cron: "50 * * * *" + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -244,38 +227,34 @@ steps below. To disable [the dependencies validation](../ci/yaml/README.md#when-a-dependent-job-will-fail), you can flip the feature flag from a Rails console. ---- - **In Omnibus installations:** 1. Enter the Rails console: - ```sh - sudo gitlab-rails console - ``` + ```sh + sudo gitlab-rails console + ``` 1. Flip the switch and disable it: - ```ruby - Feature.enable('ci_disable_validates_dependencies') - ``` - ---- + ```ruby + Feature.enable('ci_disable_validates_dependencies') + ``` **In installations from source:** 1. Enter the Rails console: - ```sh - cd /home/git/gitlab - RAILS_ENV=production sudo -u git -H bundle exec rails console - ``` + ```sh + cd /home/git/gitlab + RAILS_ENV=production sudo -u git -H bundle exec rails console + ``` 1. Flip the switch and disable it: - ```ruby - Feature.enable('ci_disable_validates_dependencies') - ``` + ```ruby + Feature.enable('ci_disable_validates_dependencies') + ``` ## Set the maximum file size of the artifacts diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md index aa9d87562a3..6a06eb240de 100644 --- a/doc/administration/job_traces.md +++ b/doc/administration/job_traces.md @@ -25,11 +25,11 @@ To change the location where the job logs will be stored, follow the steps below **In Omnibus installations:** -1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line: +1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line: - ``` - gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' - ``` + ```ruby + gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. @@ -39,12 +39,12 @@ To change the location where the job logs will be stored, follow the steps below 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - gitlab_ci: - # The location where build traces are stored (default: builds/). - # Relative paths are relative to Rails.root. - builds_path: path/to/builds/ - ``` + ```yaml + gitlab_ci: + # The location where build traces are stored (default: builds/). + # Relative paths are relative to Rails.root. + builds_path: path/to/builds/ + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -67,24 +67,24 @@ To archive those legacy job traces, please follow the instruction below. 1. Execute the following command - ```bash - gitlab-rake gitlab:traces:archive - ``` + ```bash + gitlab-rake gitlab:traces:archive + ``` - After you executed this task, GitLab instance queues up Sidekiq jobs (asynchronous processes) - for migrating job trace files from local storage to object storage. - It could take time to complete the all migration jobs. You can check the progress by the following command + After you executed this task, GitLab instance queues up Sidekiq jobs (asynchronous processes) + for migrating job trace files from local storage to object storage. + It could take time to complete the all migration jobs. You can check the progress by the following command - ```bash - sudo gitlab-rails console - ``` + ```bash + sudo gitlab-rails console + ``` - ```bash - [1] pry(main)> Sidekiq::Stats.new.queues['pipeline_background:archive_trace'] - => 100 - ``` + ```bash + [1] pry(main)> Sidekiq::Stats.new.queues['pipeline_background:archive_trace'] + => 100 + ``` - If the count becomes zero, the archiving processes are done + If the count becomes zero, the archiving processes are done ## How to migrate archived job traces to object storage @@ -95,9 +95,9 @@ If job traces have already been archived into local storage, and you want to mig 1. Ensure [Object storage integration for Job Artifacts](job_artifacts.md#object-storage-settings) is enabled 1. Execute the following command - ```bash - gitlab-rake gitlab:traces:migrate - ``` + ```bash + gitlab-rake gitlab:traces:migrate + ``` ## How to remove job traces diff --git a/doc/administration/logs.md b/doc/administration/logs.md index c5cfb8d5016..9030c941cad 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -4,7 +4,7 @@ GitLab has an advanced log system where everything is logged so that you can analyze your instance using various system log files. In addition to system log files, GitLab Enterprise Edition comes with Audit Events. Find more about them [in Audit Events -documentation](http://docs.gitlab.com/ee/administration/audit_events.html) +documentation](audit_events.md) System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. @@ -88,7 +88,7 @@ Introduced in GitLab 10.0, this file lives in It helps you see requests made directly to the API. For example: ```json -{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} +{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} ``` This entry above shows an access to an internal endpoint to check whether an @@ -125,7 +125,7 @@ This file lives in `/var/log/gitlab/gitlab-rails/integrations_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/integrations_json.log` for installations from source. -It contains information about [integrations](../user/project/integrations/project_services.md) activities such as JIRA, Asana and Irker services. It uses JSON format like the example below: +It contains information about [integrations](../user/project/integrations/project_services.md) activities such as Jira, Asana and Irker services. It uses JSON format like the example below: ``` json {"severity":"ERROR","time":"2018-09-06T14:56:20.439Z","service_class":"JiraService","project_id":8,"project_path":"h5bp/html5-boilerplate","message":"Error sending message","client_url":"http://jira.gitlap.com:8080","error":"execution expired"} @@ -151,22 +151,29 @@ etc. For example: {"severity":"ERROR","time":"2018-11-23T15:42:11.647Z","exception":"Kubeclient::HttpError","error_code":null,"service":"Clusters::Applications::InstallService","app_id":2,"project_ids":[19],"group_ids":[],"message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)"} ``` -## `githost.log` +## `git_json.log` -This file lives in `/var/log/gitlab/gitlab-rails/githost.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/githost.log` for +This file lives in `/var/log/gitlab/gitlab-rails/git_json.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/git_json.log` for installations from source. +NOTE: **Note:** +After 12.2, this file was renamed from `githost.log` to +`git_json.log` and stored in JSON format. + GitLab has to interact with Git repositories but in some rare cases something can go wrong and in this case you will know what exactly happened. This log file contains all failed requests from GitLab to Git repositories. In the majority of cases this file will be useful for developers only. For example: -``` -December 03, 2014 13:20 -> ERROR -> Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict - -error: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git' +```json +{ + "severity":"ERROR", + "time":"2019-07-19T22:16:12.528Z", + "correlation_id":"FeGxww5Hj64", + "message":"Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict\n\nerror: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git'" +} ``` ## `audit_json.log` @@ -236,7 +243,7 @@ I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory User clone/fetch activity using ssh transport appears in this log as `executing git command <gitaly-upload-pack...`. -## `unicorn\_stderr.log` +## `unicorn_stderr.log` This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for @@ -277,16 +284,19 @@ Introduced in GitLab 11.3. This file lives in `/var/log/gitlab/gitlab-rails/impo Omnibus GitLab packages or in `/home/git/gitlab/log/importer.log` for installations from source. -Currently it logs the progress of project imports from the Bitbucket Server -importer. Future importers may use this file. - -## `auth.log` +## `auth.log` Introduced in GitLab 12.0. This file lives in `/var/log/gitlab/gitlab-rails/auth.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/auth.log` for installations from source. -It logs information whenever [Rack Attack] registers an abusive request. +This log records: + +- Information whenever [Rack Attack] registers an abusive request. +- Requests over the [Rate Limit] on raw endpoints. + +NOTE: **Note:** +From [%12.1](https://gitlab.com/gitlab-org/gitlab-ce/issues/62756), user id and username are available on this log. ## `graphql_json.log` @@ -302,6 +312,12 @@ GraphQL queries are recorded in that file. For example: {"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration":7} ``` +## `migrations.log` + +Introduced in GitLab 12.3. This file lives in `/var/log/gitlab/gitlab-rails/migrations.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/migrations.log` for +installations from source. + ## Reconfigure Logs Reconfigure log files live in `/var/log/gitlab/reconfigure` for Omnibus GitLab @@ -321,3 +337,4 @@ installations from source. [repocheck]: repository_checks.md [Rack Attack]: ../security/rack_attack.md +[Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 5e9ba4f640f..0b065082ded 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -1,4 +1,4 @@ -# Merge request diffs storage **[CORE ONLY]** +# Merge request diffs storage **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/52568) in GitLab 11.8. @@ -10,7 +10,7 @@ By default, merge request diffs are stored in the database, in a table named `merge_request_diff_files`. Larger installations may find this table grows too large, in which case, switching to external storage is recommended. -### Using external storage +## Using external storage Merge request diffs can be stored on disk, or in object storage. In general, it is better to store the diffs in the database than on disk. @@ -21,18 +21,18 @@ To enable external storage of merge request diffs, follow the instructions below 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['external_diffs_enabled'] = true - ``` + ```ruby + gitlab_rails['external_diffs_enabled'] = true + ``` 1. _The external diffs will be stored in in `/var/opt/gitlab/gitlab-rails/shared/external-diffs`._ To change the path, for example, to `/mnt/storage/external-diffs`, edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['external_diffs_storage_path'] = "/mnt/storage/external-diffs" - ``` + ```ruby + gitlab_rails['external_diffs_storage_path'] = "/mnt/storage/external-diffs" + ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -41,31 +41,31 @@ To enable external storage of merge request diffs, follow the instructions below 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - external_diffs: - enabled: true - ``` + ```yaml + external_diffs: + enabled: true + ``` -1. _The external diffs will be stored in +1. _The external diffs will be stored in `/home/git/gitlab/shared/external-diffs`._ To change the path, for example, to `/mnt/storage/external-diffs`, edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - external_diffs: - enabled: true - storage_path: /mnt/storage/external-diffs - ``` + ```yaml + external_diffs: + enabled: true + storage_path: /mnt/storage/external-diffs + ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -### Using object storage +## Using object storage Instead of storing the external diffs on disk, we recommended the use of an object store like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. -### Object Storage Settings +## Object Storage Settings For source installations, these settings are nested under `external_diffs:` and then `object_store:`. On Omnibus installations, they are prefixed by @@ -80,7 +80,7 @@ then `object_store:`. On Omnibus installations, they are prefixed by | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | -#### S3 compatible connection settings +### S3 compatible connection settings The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: @@ -90,6 +90,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -101,28 +102,28 @@ The connection settings match those provided by [Fog](https://github.com/fog), a 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: - ```ruby - gitlab_rails['external_diffs_enabled'] = true - gitlab_rails['external_diffs_object_store_enabled'] = true - gitlab_rails['external_diffs_object_store_remote_directory'] = "external-diffs" - gitlab_rails['external_diffs_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', - 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' - } - ``` - - Note that, if you are using AWS IAM profiles, be sure to omit the - AWS access key and secret access key/value pairs. For example: - - ```ruby - gitlab_rails['external_diffs_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'use_iam_profile' => true - } - ``` + ```ruby + gitlab_rails['external_diffs_enabled'] = true + gitlab_rails['external_diffs_object_store_enabled'] = true + gitlab_rails['external_diffs_object_store_remote_directory'] = "external-diffs" + gitlab_rails['external_diffs_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + Note that, if you are using AWS IAM profiles, be sure to omit the + AWS access key and secret access key/value pairs. For example: + + ```ruby + gitlab_rails['external_diffs_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -131,22 +132,22 @@ The connection settings match those provided by [Fog](https://github.com/fog), a 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - external_diffs: - enabled: true - object_store: - enabled: true - remote_directory: "external-diffs" # The bucket name - connection: - provider: AWS # Only AWS supported at the moment - aws_access_key_id: AWS_ACCESS_KEY_ID - aws_secret_access_key: AWS_SECRET_ACCESS_KEY - region: eu-central-1 - ``` + ```yaml + external_diffs: + enabled: true + object_store: + enabled: true + remote_directory: "external-diffs" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -### Alternative in-database storage +## Alternative in-database storage Enabling external diffs may reduce the performance of merge requests, as they must be retrieved in a separate operation to other data. A compromise may be @@ -157,11 +158,11 @@ To enable this feature, perform the following steps: **In Omnibus installations:** -1. Edit `/etc/gitlab/gitlab.rb` and add the following line: +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['external_diffs_when'] = 'outdated' - ``` + ```ruby + gitlab_rails['external_diffs_when'] = 'outdated' + ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -170,11 +171,11 @@ To enable this feature, perform the following steps: 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - external_diffs: - enabled: true - when: outdated - ``` + ```yaml + external_diffs: + enabled: true + when: outdated + ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/monitoring/gitlab_instance_administration_project/index.md b/doc/administration/monitoring/gitlab_instance_administration_project/index.md new file mode 100644 index 00000000000..8e33cea6217 --- /dev/null +++ b/doc/administration/monitoring/gitlab_instance_administration_project/index.md @@ -0,0 +1,36 @@ +# GitLab instance administration project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/56883) in GitLab 12.2. + +GitLab has been adding the ability for administrators to see insights into the health of +their GitLab instance. In order to surface this experience in a native way, similar to how +you would interact with an application deployed via GitLab, a base project called +"GitLab Instance Administration" with +[internal visibility](../../../public_access/public_access.md#internal-projects) will be +added under a group called "GitLab Instance Administrators" specifically created for +visualizing and configuring the monitoring of your GitLab instance. + +All administrators at the time of creation of the project and group will be added +as maintainers of the group and project, and as an admin, you'll be able to add new +members to the group in order to give them maintainer access to the project. + +This project will be used for self-monitoring your GitLab instance. + +## Connection to Prometheus + +The project will be automatically configured to connect to the +[internal Prometheus](../prometheus/index.md) instance if the Prometheus +instance is present (should be the case if GitLab was installed via Omnibus +and you haven't disabled it). + +If that's not the case or if you have an external Prometheus instance or an HA setup, +you should +[configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). + +## Taking action on Prometheus alerts **[ULTIMATE]** + +You can [add a webhook](../../../user/project/integrations/prometheus.md#external-prometheus-instances) +to the Prometheus config in order for GitLab to receive notifications of any alerts. + +Once the webhook is setup, you can +[take action on incoming alerts](../../../user/project/integrations/prometheus.md#taking-action-on-incidents-ultimate). diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index fa0459b24ff..2b3daec42bd 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -2,6 +2,9 @@ Explore our features to monitor your GitLab instance: +- [GitLab self-monitoring](gitlab_instance_administration_project/index.md): The + GitLab instance administration project helps to monitor the GitLab instance and + take action on alerts. - [Performance monitoring](performance/index.md): GitLab Performance Monitoring makes it possible to measure a wide variety of statistics of your instance. - [Prometheus](prometheus/index.md): Prometheus is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. - [GitHub imports](github_imports.md): Monitor the health and progress of GitLab's GitHub importer with various Prometheus metrics. diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index ad2773de132..6bb2fe81b2c 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -12,9 +12,9 @@ hosts or use IP ranges: 1. Open `/etc/gitlab/gitlab.rb` and add or uncomment the following: - ```ruby - gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] - ``` + ```ruby + gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] + ``` 1. Save the file and [reconfigure] GitLab for the changes to take effect. @@ -24,13 +24,13 @@ hosts or use IP ranges: 1. Edit `config/gitlab.yml`: - ```yaml - monitoring: - # by default only local IPs are allowed to access monitoring resources - ip_whitelist: - - 127.0.0.0/8 - - 192.168.0.1 - ``` + ```yaml + monitoring: + # by default only local IPs are allowed to access monitoring resources + ip_whitelist: + - 127.0.0.0/8 + - 192.168.0.1 + ``` 1. Save the file and [restart] GitLab for the changes to take effect. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 771584268d9..9a25cc04ee7 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -8,12 +8,8 @@ The minimum required settings you need to set are the InfluxDB host and port. Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the changes. ---- -  ---- - Finally, a restart of all GitLab processes is required for the changes to take effect: @@ -30,8 +26,6 @@ sudo service gitlab restart When any migrations are pending, the metrics are disabled until the migrations have been performed. ---- - Read more on: - [Introduction to GitLab Performance Monitoring](introduction.md) diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 187fb2f73a1..95be0d5fd88 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,6 +1,6 @@ # Grafana Configuration -[Grafana](http://grafana.org/) is a tool that allows you to visualize time +[Grafana](https://grafana.com/) is a tool that allows you to visualize time series metrics through graphs and dashboards. It supports several backend data stores, including InfluxDB. GitLab writes performance data to InfluxDB and Grafana will allow you to query to display useful graphs. @@ -13,7 +13,7 @@ services. [GitLab Omnibus can help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) or Grafana supplies package repositories (Yum/Apt) for easy installation. -See [Grafana installation documentation](http://docs.grafana.org/installation/) +See [Grafana installation documentation](https://grafana.com/docs/installation/) for detailed steps. NOTE: **Note:** @@ -103,6 +103,51 @@ repository for more information on this process. [grafana-dashboards]: https://gitlab.com/gitlab-org/grafana-dashboards +## Integration with GitLab UI + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/61005) in GitLab 12.1. + +If you have set up Grafana, you can enable a link to access it easily from the sidebar: + +1. Go to the admin area under **Settings > Metrics and profiling** + and expand "Metrics - Grafana". +1. Check the "Enable access to Grafana" checkbox. +1. If Grafana is enabled through Omnibus GitLab and on the same server, + leave "Grafana URL" unchanged. In any other case, enter the full URL + path of the Grafana instance. +1. Click **Save changes**. +1. The new link will be available in the admin area under **Monitoring > Metrics Dashboard**. + +## Security Update + +Users running GitLab version 12.0 or later should immediately upgrade to one of the following security releases due to a known vulnerability with the embedded Grafana dashboard: + +- 12.0.6 +- 12.1.6 + +After upgrading, the Grafana dashboard will be disabled and the location of your existing Grafana data will be changed from `/var/opt/gitlab/grafana/data/` to `/var/opt/gitlab/grafana/data.bak.#{Date.today}/`. + +To prevent the data from being relocated, you can run the following command prior to upgrading: + +```sh +echo "0" > /var/opt/gitlab/grafana/CVE_reset_status +``` + +To reinstate your old data, move it back into its original location: + +``` +sudo mv /var/opt/gitlab/grafana/data.bak.xxxx/ /var/opt/gitlab/grafana/data/ +``` + +However, you should **not** reinstate your old data _except_ under one of the following conditions: + +1. If you are certain that you changed your default admin password when you enabled Grafana +1. If you run GitLab in a private network, accessed only by trusted users, and your Grafana login page has not been exposed to the internet + +If you require access to your old Grafana data but do not meet one of these criteria, you may consider reinstating it temporarily, [exporting the dashboards](https://grafana.com/docs/reference/export_import/#exporting-a-dashboard) you need, then refreshing the data and [re-importing your dashboards](https://grafana.com/docs/reference/export_import/#importing-a-dashboard). Note that this poses a temporary vulnerability while your old Grafana data is in use, and the decision to do so should be weighed carefully with your need to access existing data and dashboards. + +For more information and further mitigation details, please refer to our [blog post on the security release](https://about.gitlab.com/2019/08/12/critical-security-release-gitlab-12-dot-1-dot-6-released/). + --- Read more on: diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differindex 2bf686f9017..d1187fd879a 100644 --- a/doc/administration/monitoring/performance/img/performance_bar.png +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png Binary files differindex 7af6d401d1d..2c43201cbd0 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png +++ b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png Binary files differdeleted file mode 100644 index a55ce753101..00000000000 --- a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png b/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png Binary files differnew file mode 100644 index 00000000000..ecb2dffbf8d --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png b/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png Binary files differnew file mode 100644 index 00000000000..210f80a713d --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png Binary files differindex b3219b4fa94..6b571f4e85c 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png +++ b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png diff --git a/doc/administration/monitoring/performance/img/request_profile_result.png b/doc/administration/monitoring/performance/img/request_profile_result.png Binary files differindex 1b06e240fa0..9176a0b49fd 100644 --- a/doc/administration/monitoring/performance/img/request_profile_result.png +++ b/doc/administration/monitoring/performance/img/request_profile_result.png diff --git a/doc/administration/monitoring/performance/influxdb_configuration.md b/doc/administration/monitoring/performance/influxdb_configuration.md index fa281f47ed8..cf6728510fe 100644 --- a/doc/administration/monitoring/performance/influxdb_configuration.md +++ b/doc/administration/monitoring/performance/influxdb_configuration.md @@ -187,7 +187,7 @@ Read more on: [influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ [influxdb cli]: https://docs.influxdata.com/influxdb/v0.9/tools/shell/ [udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ -[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[influxdb]: https://www.influxdata.com/products/influxdb-overview/ [tsm tree]: https://influxdata.com/blog/new-storage-engine-time-structured-merge-tree/ [tsm1-commit]: https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d [influx-admin]: https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 95f497a1470..02f4b78bd60 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -8,16 +8,14 @@ activated, it looks as follows: It allows you to see (from left to right): - the current host serving the page -- the timing of the page (backend, frontend) - time taken and number of DB queries, click through for details of these queries  - time taken and number of [Gitaly] calls, click through for details of these calls  -- profile of the code used to generate the page, line by line. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)). -  -- time taken and number of calls to Redis -- time taken and number of background jobs created by Sidekiq -- time taken and number of Ruby GC calls +- time taken and number of [Rugged] calls, click through for details of these calls +  +- time taken and number of Redis calls, click through for details of these calls +  On the far right is a request selector that allows you to view the same metrics (excluding the page timing and line profiler) for any requests made while the @@ -40,10 +38,7 @@ display it. You can toggle the Bar using the same shortcut. ---- -  ---- - [Gitaly]: ../../gitaly/index.md +[Rugged]: ../../high_availability/nfs.md#improving-nfs-performance-with-gitlab diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 726882fbb87..9f671e0db11 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -5,9 +5,9 @@ 1. Grab the profiling token from **Monitoring > Requests Profiles** admin page (highlighted in a blue in the image below).  -1. Pass the header `X-Profile-Token: <token>` to the request you want to profile. You can use: +1. Pass the header `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where <mode> can be `execution` or `memory`) to the request you want to profile. You can use: - Browser extensions. For example, [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension. - - `curl`. For example, `curl --header 'X-Profile-Token: <token>' https://gitlab.example.com/group/project`. + - `curl`. For example, `curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' https://gitlab.example.com/group/project`. 1. Once request is finished (which will take a little longer than usual), you can view the profiling output from **Monitoring > Requests Profiles** admin page.  diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 84b71ae6f1c..ec26c0b2e7e 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -34,6 +34,9 @@ The following metrics are available: | filesystem_writable | Gauge | 9.4 | Whether or not the filesystem is writable | | filesystem_read_latency_seconds | Gauge | 9.4 | Read latency of a specific filesystem | | filesystem_readable | Gauge | 9.4 | Whether or not the filesystem is readable | +| gitlab_cache_misses_total | Counter | 10.2 | Cache read miss | +| gitlab_cache_operation_duration_seconds | Histogram | 10.2 | Cache access time | +| gitlab_cache_operations_total | Counter | 12.2 | Cache operations by controller/action | | http_requests_total | Counter | 9.4 | Rack request count | | http_request_duration_seconds | Histogram | 9.4 | HTTP response time from rack middleware | | pipelines_created_total | Counter | 9.4 | Counter of pipelines created | @@ -49,7 +52,7 @@ The following metrics are available: | unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | | unicorn_workers | Gauge | 12.0 | The number of Unicorn workers | -## Sidekiq Metrics available for Geo **[PREMIUM]** +## Sidekiq Metrics available for Geo **(PREMIUM)** Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled (e.g. via the `monitoring.sidekiq_exporter` configuration option in `gitlab.yml`. @@ -104,11 +107,11 @@ Some basic Ruby runtime metrics are available: | ruby_process_cpu_seconds_total | Gauge | 12.0 | Total amount of CPU time per process | | ruby_process_max_fds | Gauge | 12.0 | Maximum number of open file descriptors per process | | ruby_process_resident_memory_bytes | Gauge | 12.0 | Memory usage by process, measured in bytes | -| ruby_process_start_time_seconds | Gauge | 12.0 | The elapsed time between system boot and the process started, measured in seconds | +| ruby_process_start_time_seconds | Gauge | 12.0 | UNIX timestamp of process start time | [GC.stat]: https://ruby-doc.org/core-2.3.0/GC.html#method-c-stat -## Puma Metrics **[EXPERIMENTAL]** +## Puma Metrics **(EXPERIMENTAL)** When Puma is used instead of Unicorn, following metrics are available: @@ -117,7 +120,6 @@ When Puma is used instead of Unicorn, following metrics are available: | puma_workers | Gauge | 12.0 | Total number of workers | | puma_running_workers | Gauge | 12.0 | Number of booted workers | | puma_stale_workers | Gauge | 12.0 | Number of old workers | -| puma_phase | Gauge | 12.0 | Phase number (increased during phased restarts) | | puma_running | Gauge | 12.0 | Number of running threads | | puma_queued_connections | Gauge | 12.0 | Number of connections in that worker's "todo" set waiting for a worker thread | | puma_active_connections | Gauge | 12.0 | Number of threads processing a request | diff --git a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md index f68b03d1ade..9aa4dfa5ab7 100644 --- a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md @@ -12,9 +12,9 @@ To enable the GitLab monitor exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - gitlab_monitor['enable'] = true - ``` + ```ruby + gitlab_monitor['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index ce65d77274b..c8968c51393 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -39,9 +39,9 @@ To disable Prometheus and all of its exporters, as well as any added in the futu 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `false`: - ```ruby - prometheus_monitoring['enable'] = false - ``` + ```ruby + prometheus_monitoring['enable'] = false + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect. @@ -61,19 +61,19 @@ To change the address/port that Prometheus listens on: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line: - ```ruby - prometheus['listen_address'] = 'localhost:9090' - ``` + ```ruby + prometheus['listen_address'] = 'localhost:9090' + ``` - Replace `localhost:9090` with the address/port you want Prometheus to - listen on. If you would like to allow access to Prometheus to hosts other - than `localhost`, leave out the host, or use `0.0.0.0` to allow public access: + Replace `localhost:9090` with the address/port you want Prometheus to + listen on. If you would like to allow access to Prometheus to hosts other + than `localhost`, leave out the host, or use `0.0.0.0` to allow public access: - ```ruby - prometheus['listen_address'] = ':9090' - # or - prometheus['listen_address'] = '0.0.0.0:9090' - ``` + ```ruby + prometheus['listen_address'] = ':9090' + # or + prometheus['listen_address'] = '0.0.0.0:9090' + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect @@ -90,22 +90,22 @@ To use an external Prometheus server: 1. Edit `/etc/gitlab/gitlab.rb`. 1. Disable the bundled Prometheus: - ```ruby - prometheus['enable'] = false - ``` + ```ruby + prometheus['enable'] = false + ``` 1. Set each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, for example: - ```ruby - gitlab_monitor['listen_address'] = '0.0.0.0' - sidekiq['listen_address'] = '0.0.0.0' - gitlab_monitor['listen_port'] = '9168' - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229" - ``` + ```ruby + gitlab_monitor['listen_address'] = '0.0.0.0' + sidekiq['listen_address'] = '0.0.0.0' + gitlab_monitor['listen_port'] = '9168' + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229" + ``` 1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/). 1. Add the Prometheus server IP address to the [monitoring IP whitelist](../ip_whitelist.html). For example: @@ -117,14 +117,14 @@ To use an external Prometheus server: 1. To scrape nginx metrics, you'll also need to configure nginx to allow the Prometheus server IP. For example: - ```ruby - nginx['status']['options'] = { - "server_tokens" => "off", - "access_log" => "off", - "allow" => "192.168.0.1", - "deny" => "all", - } - ``` + ```ruby + nginx['status']['options'] = { + "server_tokens" => "off", + "access_log" => "off", + "allow" => "192.168.0.1", + "deny" => "all", + } + ``` 1. [Reconfigure GitLab][reconfigure] to apply the changes 1. Edit the Prometheus server's configuration file. @@ -132,19 +132,59 @@ To use an external Prometheus server: [scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E). For example, a sample snippet using `static_configs`: - ```yaml - scrape_configs: - - job_name: 'gitlab_exporters' - static_configs: - - targets: ['1.1.1.1:9168', '1.1.1.1:9236', '1.1.1.1:9236', '1.1.1.1:9100', '1.1.1.1:9121', '1.1.1.1:9187'] - - - job_name: 'gitlab_metrics' - metrics_path: /-/metrics - static_configs: - - targets: ['1.1.1.1:443'] - ``` - -1. Restart the Prometheus server. + ```yaml + scrape_configs: + - job_name: nginx + static_configs: + - targets: + - 1.1.1.1:8060 + - job_name: redis + static_configs: + - targets: + - 1.1.1.1:9121 + - job_name: postgres + static_configs: + - targets: + - 1.1.1.1:9187 + - job_name: node + static_configs: + - targets: + - 1.1.1.1:9100 + - job_name: gitlab-workhorse + static_configs: + - targets: + - 1.1.1.1:9229 + - job_name: gitlab-rails + metrics_path: "/-/metrics" + static_configs: + - targets: + - 1.1.1.1:8080 + - job_name: gitlab-sidekiq + static_configs: + - targets: + - 1.1.1.1:8082 + - job_name: gitlab_monitor_database + metrics_path: "/database" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitlab_monitor_sidekiq + metrics_path: "/sidekiq" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitlab_monitor_process + metrics_path: "/process" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitaly + static_configs: + - targets: + - 1.1.1.1:9236 + ``` + +1. Reload the Prometheus server. ## Viewing performance metrics @@ -241,9 +281,9 @@ To disable the monitoring of Kubernetes: 1. Edit `/etc/gitlab/gitlab.rb`. 1. Add or find and uncomment the following line and set it to `false`: - ```ruby - prometheus['monitor_kubernetes'] = false - ``` + ```ruby + prometheus['monitor_kubernetes'] = false + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect. diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index aef7758a88f..bcacfaa3be5 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -13,9 +13,9 @@ To enable the node exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - node_exporter['enable'] = true - ``` + ```ruby + node_exporter['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index d76834fdbea..6183594c69c 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -12,9 +12,9 @@ To enable the PgBouncer exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - pgbouncer_exporter['enable'] = true - ``` + ```ruby + pgbouncer_exporter['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect. diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 8e2d3162f88..3ad15b65497 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -12,9 +12,9 @@ To enable the postgres exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - postgres_exporter['enable'] = true - ``` + ```ruby + postgres_exporter['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index d54d409dbb6..1520115a38d 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -13,9 +13,9 @@ To enable the Redis exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - redis_exporter['enable'] = true - ``` + ```ruby + redis_exporter['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 286b99aceb5..a16cd5166b7 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,70 +1,132 @@ -# Extra Sidekiq processes **[STARTER ONLY]** +# Extra Sidekiq processes **(STARTER ONLY)** -GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes +NOTE: **Note:** +The information in this page applies only to Omnibus GitLab. + +GitLab Starter allows one to start an extra set of Sidekiq processes besides the default one. These processes can be used to consume a dedicated set of queues. This can be used to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. -## Starting extra processes via Omnibus GitLab +## Available Sidekiq queues -To enable `sidekiq-cluster`, you must apply the `sidekiq_cluster['enable'] = true` -setting `/etc/gitlab/gitlab.rb`: +For a list of the existing Sidekiq queues, check the following files: -```ruby -sidekiq_cluster['enable'] = true -``` +- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/workers/all_queues.yml) +- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/workers/all_queues.yml) -You will then specify how many additional processes to create via `sidekiq-cluster` -as well as which queues for them to handle. This is done via the -`sidekiq_cluster['queue_groups']` setting. This is an array whose items contain -which queues to process. Each item in the array will equate to one additional -sidekiq process. +Each entry in the above files represents a queue on which extra Sidekiq processes +can be started. -As an example, to make additional sidekiq processes that process the -`elastic_indexer` and `mailers` queues, you would apply the following: +## Starting extra processes -```ruby -sidekiq_cluster['queue_groups'] = [ - "elastic_indexer", - "mailers" -] -``` +To start extra Sidekiq processes, you must enable `sidekiq-cluster`: -To have an additional sidekiq process handle multiple queues, you simply put a -comma after the first queue name and then put the next queue name: +1. Edit `/etc/gitlab/gitlab.rb` and add: -```ruby -sidekiq_cluster['queue_groups'] = [ - "elastic_indexer,elastic_commit_indexer", - "mailers" -] -``` + ```ruby + sidekiq_cluster['enable'] = true + ``` -Keep in mind, all changes must be followed by reconfiguring your GitLab -application via `sudo gitlab-ctl reconfigure`. +1. You will then need to specify how many additional processes to create via `sidekiq-cluster` + and which queue they should handle via the `sidekiq_cluster['queue_groups']` + array setting. Each item in the array equates to one additional Sidekiq + process, and values in each item determine the queues it works on. -### Monitoring + For example, the following setting adds additional Sidekiq processes to two + queues, one to `elastic_indexer` and one to `mailers`: -Once the Sidekiq processes are added, you can visit the "Background Jobs" + ```ruby + sidekiq_cluster['queue_groups'] = [ + "elastic_indexer", + "mailers" + ] + ``` + + To have an additional Sidekiq process handle multiple queues, add multiple + queue names to its item delimited by commas. For example: + + ```ruby + sidekiq_cluster['queue_groups'] = [ + "elastic_indexer, elastic_commit_indexer", + "mailers" + ] + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +Once the extra Sidekiq processes are added, you can visit the "Background Jobs" section under the admin area in GitLab (`/admin/background_jobs`). - + -### All queues with exceptions +## Negating settings -To have the additional sidekiq processes work on every queue EXCEPT the ones +To have the additional Sidekiq processes work on every queue **except** the ones you list: +1. After you follow the steps for [starting extra processes](#starting-extra-processes), + edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['negate'] = true + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +## Ignore all GitHub import queues + +When [importing from GitHub](../../user/project/import/github.md), Sidekiq might +use all of its resources to perform those operations. To set up a separate +`sidekiq-cluster` process to ignore all GitHub import-related queues: + 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby + sidekiq_cluster['enable'] = true sidekiq_cluster['negate'] = true + sidekiq_cluster['queue_groups'] = [ + "github_import_advance_stage", + "github_importer:github_import_import_diff_note", + "github_importer:github_import_import_issue", + "github_importer:github_import_import_note", + "github_importer:github_import_import_lfs_object", + "github_importer:github_import_import_pull_request", + "github_importer:github_import_refresh_import_jid", + "github_importer:github_import_stage_finish_import", + "github_importer:github_import_stage_import_base_data", + "github_importer:github_import_stage_import_issues_and_diff_notes", + "github_importer:github_import_stage_import_notes", + "github_importer:github_import_stage_import_lfs_objects", + "github_importer:github_import_stage_import_pull_requests", + "github_importer:github_import_stage_import_repository" + ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab for the changes to take effect: + ```sh + sudo gitlab-ctl reconfigure + ``` -### Limiting concurrency +## Number of threads + +Each process defined under `sidekiq_cluster` starts with a +number of threads that equals the number of queues, plus one spare thread. +For example, a process that handles the `process_commit` and `post_receive` +queues will use three threads in total. + +## Limiting concurrency + +To limit the concurrency of the Sidekiq processes: 1. Edit `/etc/gitlab/gitlab.rb` and add: @@ -72,11 +134,22 @@ you list: sidekiq_cluster['concurrency'] = 25 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab for the changes to take effect: -Keep in mind, this normally would not exceed the number of CPU cores available. + ```sh + sudo gitlab-ctl reconfigure + ``` -### Modifying the check interval +For each queue group, the concurrency factor will be set to `min(number of queues, N)`. +Setting the value to 0 will disable the limit. Keep in mind this normally would +not exceed the number of CPU cores available. + +Each thread requires a Redis connection, so adding threads may +increase Redis latency and potentially cause client timeouts. See the [Sidekiq +documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) +for more details. + +## Modifying the check interval To modify the check interval for the additional Sidekiq processes: @@ -90,9 +163,14 @@ To modify the check interval for the additional Sidekiq processes: This tells the additional processes how often to check for enqueued jobs. -## Starting extra processes via command line +## Troubleshooting using the CLI -Starting extra Sidekiq processes can be done using the command +CAUTION: **Warning:** +It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes. +If you experience a problem, you should contact GitLab support. Use the command +line at your own risk. + +For debugging purposes, you can start extra Sidekiq processes by using the command `/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster`. This command takes arguments using the following syntax: @@ -111,29 +189,29 @@ see the relevant section in the [Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces). For example, say you want to start 2 extra processes: one to process the -"process_commit" queue, and one to process the "post_receive" queue. This can be +`process_commit` queue, and one to process the `post_receive` queue. This can be done as follows: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit post_receive ``` -If you instead want to start one process processing both queues you'd use the +If you instead want to start one process processing both queues, you'd use the following syntax: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive ``` -If you want to have one Sidekiq process process the "process_commit" and -"post_receive" queues, and one process to process the "gitlab_shell" queue, +If you want to have one Sidekiq process dealing with the `process_commit` and +`post_receive` queues, and one process to process the `gitlab_shell` queue, you'd use the following: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive gitlab_shell ``` -### Monitoring +### Monitoring the `sidekiq-cluster` command The `sidekiq-cluster` command will not terminate once it has started the desired amount of Sidekiq processes. Instead, the process will continue running and @@ -172,24 +250,24 @@ command and not the PID(s) of the started Sidekiq processes. The Rails environment can be set by passing the `--environment` flag to the `sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The -default value is "development". +default value can be found in `/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV`. -### All queues with exceptions +### Using negation You're able to run all queues in `sidekiq_queues.yml` file on a single or multiple processes with exceptions using the `--negate` flag. For example, say you want to run a single process for all queues, -except "process_commit" and "post_receive". You can do so by executing: +except `process_commit` and `post_receive`: ```bash -sidekiq-cluster process_commit,post_receive --negate +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive --negate ``` -For multiple processes of all queues (except "process_commit" and "post_receive"): +For multiple processes of all queues (except `process_commit` and `post_receive`): ```bash -sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate ``` ### Limiting concurrency @@ -201,18 +279,3 @@ the `-m N` option. For example, this would cap the maximum number of threads to ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive -m 1 ``` - -For each queue group, the concurrency factor will be set to min(number of -queues, N). Setting the value to 0 will disable the limit. - -Note that each thread requires a Redis connection, so adding threads may -increase Redis latency and potentially cause client timeouts. See the [Sidekiq -documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) -for more details. - -## Number of threads - -Each process started using `sidekiq-cluster` (whether it be via command line or -via the gitlab.rb file) starts with a number of threads that equals the number -of queues, plus one spare thread. For example, a process that handles the -"process_commit" and "post_receive" queues will use 3 threads in total. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 3631ea0822f..e787af798bc 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -6,7 +6,7 @@ using [ssh certificates](ssh_certificates.md), they are even faster, but are not a drop-in replacement. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1631) in -> [GitLab Starter](https://about.gitlab.com/gitlab-ee) 9.3. +> [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. > > [Available in](https://gitlab.com/gitlab-org/gitlab-ee/issues/3953) GitLab > Community Edition 10.4. @@ -30,7 +30,7 @@ instructions will break installations using older versions of OpenSSH, such as those included with CentOS 6 as of September 2017. If you want to use this feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing. -## Fast lookup is required for Geo **[PREMIUM]** +## Fast lookup is required for Geo **(PREMIUM)** By default, GitLab manages an `authorized_keys` file, which contains all the public SSH keys for users allowed to access GitLab. However, to maintain a @@ -71,10 +71,10 @@ sudo service sshd reload Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo. -> **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in +NOTE: **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in GitLab 11.11 and later. -> **Warning:** Do not disable writes until SSH is confirmed to be working +CAUTION: **Caution:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. In the case of lookup failures (which are common), the `authorized_keys` @@ -117,81 +117,81 @@ the database. The following instructions can be used to build OpenSSH 7.5: 1. First, download the package and install the required packages: - ``` - sudo su - - cd /tmp - curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz - tar xzvf openssh-7.5p1.tar.gz - yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel - ``` + ``` + sudo su - + cd /tmp + curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz + tar xzvf openssh-7.5p1.tar.gz + yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel + ``` 1. Prepare the build by copying files to the right place: - ``` - mkdir -p /root/rpmbuild/{SOURCES,SPECS} - cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/ - cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/ - cd /root/rpmbuild/SPECS - ``` + ``` + mkdir -p /root/rpmbuild/{SOURCES,SPECS} + cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/ + cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/ + cd /root/rpmbuild/SPECS + ``` 1. Next, set the spec settings properly: - ``` - sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec - sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec - sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec - ``` + ``` + sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec + sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec + sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec + ``` 1. Build the RPMs: - ``` - rpmbuild -bb openssh.spec - ``` + ``` + rpmbuild -bb openssh.spec + ``` 1. Ensure the RPMs were built: - ``` - ls -al /root/rpmbuild/RPMS/x86_64/ - ``` + ``` + ls -al /root/rpmbuild/RPMS/x86_64/ + ``` - You should see something as the following: + You should see something as the following: - ``` - total 1324 - drwxr-xr-x. 2 root root 4096 Jun 20 19:37 . - drwxr-xr-x. 3 root root 19 Jun 20 19:37 .. - -rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm - ``` + ``` + total 1324 + drwxr-xr-x. 2 root root 4096 Jun 20 19:37 . + drwxr-xr-x. 3 root root 19 Jun 20 19:37 .. + -rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm + -rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm + -rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm + -rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm + ``` 1. Install the packages. OpenSSH packages will replace `/etc/pam.d/sshd` with its own version, which may prevent users from logging in, so be sure that the file is backed up and restored after installation: - ``` - timestamp=$(date +%s) - cp /etc/pam.d/sshd pam-ssh-conf-$timestamp - rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm - yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd - ``` + ``` + timestamp=$(date +%s) + cp /etc/pam.d/sshd pam-ssh-conf-$timestamp + rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm + yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd + ``` 1. Verify the installed version. In another window, attempt to login to the server: - ``` - ssh -v <your-centos-machine> - ``` + ``` + ssh -v <your-centos-machine> + ``` - You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5" + You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5" - If not, you may need to restart sshd (e.g. `systemctl restart sshd.service`). + If not, you may need to restart sshd (e.g. `systemctl restart sshd.service`). -1. *IMPORTANT!* Open a new SSH session to your server before exiting to make - sure everything is working! If you need to downgrade, simple install the - older package: +1. *IMPORTANT!* Open a new SSH session to your server before exiting to make + sure everything is working! If you need to downgrade, simple install the + older package: - ``` - # Only run this if you run into a problem logging in - yum downgrade openssh-server openssh openssh-clients - ``` + ``` + # Only run this if you run into a problem logging in + yum downgrade openssh-server openssh openssh-clients + ``` diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index c0c242733a2..b5922d9d99d 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -78,34 +78,37 @@ executed, and then read the same 1,000 files. [repository storage path](../repository_storage_paths.md). 1. Create a temporary directory for the test so it's easy to remove the files later: - ```sh - mkdir test; cd test - ``` + ```sh + mkdir test; cd test + ``` + 1. Run the command: - ```sh - time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done - ``` -1. To benchmark read performance, run the command: + ```sh + time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done + ``` - ```sh - time for i in {0..1000}; do cat "test${i}.txt" > /dev/null; done - ``` -1. Remove the test files: +1. To benchmark read performance, run the command: ```sh - cd ../; rm -rf test + time for i in {0..1000}; do cat "test${i}.txt" > /dev/null; done ``` +1. Remove the test files: + + ```sh + cd ../; rm -rf test + ``` + The output of the `time for ...` commands will look similar to the following. The important metric is the `real` time. ```sh $ time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done -real 0m0.116s -user 0m0.025s -sys 0m0.091s +real 0m0.116s +user 0m0.025s +sys 0m0.091s $ time for i in {0..1000}; do cat "test${i}.txt" > /dev/null; done diff --git a/doc/administration/operations/img/sidekiq-cluster.png b/doc/administration/operations/img/sidekiq-cluster.png Binary files differindex 4eb1849010e..3899385eb8f 100644 --- a/doc/administration/operations/img/sidekiq-cluster.png +++ b/doc/administration/operations/img/sidekiq-cluster.png diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index df795a48169..df208b3f427 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -11,7 +11,7 @@ Keep your GitLab instance up and running smoothly. by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. -- [Extra Sidekiq operations](extra_sidekiq_processes.md): Configure an extra set of Sidekiq processes to ensure certain queues always have dedicated workers, no matter the amount of jobs that need to be processed. **[STARTER ONLY]** +- [Extra Sidekiq operations](extra_sidekiq_processes.md): Configure an extra set of Sidekiq processes to ensure certain queues always have dedicated workers, no matter the amount of jobs that need to be processed. **(STARTER ONLY)** - [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index 0e2079cb093..8178cb243f3 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -2,7 +2,7 @@ ## Unicorn -GitLab uses [Unicorn](http://unicorn.bogomips.org/), a pre-forking Ruby web +GitLab uses [Unicorn](https://bogomips.org/unicorn/), a pre-forking Ruby web server, to handle web requests (web browsers and Git HTTP clients). Unicorn is a daemon written in Ruby and C that can load and run a Ruby on Rails application; in our case the Rails application is GitLab Community Edition or @@ -69,7 +69,7 @@ unicorn['worker_memory_limit_min'] = "400 * 1 << 20" unicorn['worker_memory_limit_max'] = "650 * 1 << 20" ``` -Otherwise, you can set the `GITLAB_UNICORN_MEMORY_MIN` and `GITLAB_UNICORN_MEMORY_MIN` +Otherwise, you can set the `GITLAB_UNICORN_MEMORY_MIN` and `GITLAB_UNICORN_MEMORY_MAX` [environment variables](../environment_variables.md). This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. diff --git a/doc/administration/packages.md b/doc/administration/packages.md index 0d5f784b71e..c0f8777a8c0 100644 --- a/doc/administration/packages.md +++ b/doc/administration/packages.md @@ -1,4 +1,4 @@ -# GitLab Packages administration **[PREMIUM ONLY]** +# GitLab Packages administration **(PREMIUM ONLY)** GitLab Packages allows organizations to utilize GitLab as a private repository for a variety of common package managers. Users are able to build and publish @@ -11,7 +11,7 @@ The Packages feature allows GitLab to act as a repository for the following: | [Maven Repository](../user/project/packages/maven_repository.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | | [NPM Registry](../user/project/packages/npm_registry.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | -Don't you see your package management system supported yet? +Don't you see your package management system supported yet? Please consider contributing to GitLab. This [development documentation](../development/packages.md) will guide you through the process. @@ -28,9 +28,9 @@ To enable the Packages feature: 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['packages_enabled'] = true - ``` + ```ruby + gitlab_rails['packages_enabled'] = true + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. @@ -39,10 +39,11 @@ To enable the Packages feature: 1. After the installation is complete, you will have to configure the `packages` section in `config/gitlab.yml`. Set to `true` to enable it: - ```yaml - packages: - enabled: true - ``` + ```yaml + packages: + enabled: true + ``` + 1. [Restart GitLab] for the changes to take effect. ## Changing the storage path @@ -61,9 +62,9 @@ To change the local storage path: 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['packages_storage_path'] = "/mnt/packages" - ``` + ```ruby + gitlab_rails['packages_storage_path'] = "/mnt/packages" + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. @@ -71,11 +72,12 @@ To change the local storage path: 1. Edit the `packages` section in `config/gitlab.yml`: - ```yaml - packages: - enabled: true - storage_path: shared/packages - ``` + ```yaml + packages: + enabled: true + storage_path: shared/packages + ``` + 1. [Restart GitLab] for the changes to take effect. ### Using object storage @@ -88,31 +90,31 @@ upload packages: 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where necessary): - ```ruby - gitlab_rails['packages_enabled'] = true - gitlab_rails['packages_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/packages" - gitlab_rails['packages_object_store_enabled'] = true - gitlab_rails['packages_object_store_remote_directory'] = "packages" # The bucket name. - gitlab_rails['packages_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - gitlab_rails['packages_object_store_background_upload'] = true # Temporary option to limit automatic upload (Default: true). - gitlab_rails['packages_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. - gitlab_rails['packages_object_store_connection'] = { - ## - ## If the provider is AWS S3, uncomment the following - ## - #'provider' => 'AWS', - #'region' => 'eu-west-1', - #'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', - #'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY', - ## - ## If the provider is other than AWS (an S3-compatible one), uncomment the following - ## - #'host' => 's3.amazonaws.com', - #'aws_signature_version' => 4 # For creation of signed URLs. Set to 2 if provider does not support v4. - #'endpoint' => 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. - #'path_style' => false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. - } - ``` + ```ruby + gitlab_rails['packages_enabled'] = true + gitlab_rails['packages_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/packages" + gitlab_rails['packages_object_store_enabled'] = true + gitlab_rails['packages_object_store_remote_directory'] = "packages" # The bucket name. + gitlab_rails['packages_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + gitlab_rails['packages_object_store_background_upload'] = true # Temporary option to limit automatic upload (Default: true). + gitlab_rails['packages_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + gitlab_rails['packages_object_store_connection'] = { + ## + ## If the provider is AWS S3, uncomment the following + ## + #'provider' => 'AWS', + #'region' => 'eu-west-1', + #'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + #'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY', + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #'host' => 's3.amazonaws.com', + #'aws_signature_version' => 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #'endpoint' => 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #'path_style' => false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + } + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. @@ -120,35 +122,35 @@ upload packages: 1. Edit the `packages` section in `config/gitlab.yml` (uncomment where necessary): - ```yaml - packages: - enabled: true - ## - ## The location where build packages are stored (default: shared/packages). - ## - #storage_path: shared/packages - object_store: - enabled: false - remote_directory: packages # The bucket name. - #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - #background_upload: true # Temporary option to limit automatic upload (Default: true). - #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. - connection: - ## - ## If the provider is AWS S3, uncomment the following - ## - #provider: AWS - #region: us-east-1 - #aws_access_key_id: AWS_ACCESS_KEY_ID - #aws_secret_access_key: AWS_SECRET_ACCESS_KEY - ## - ## If the provider is other than AWS (an S3-compatible one), uncomment the following - ## - #host: 's3.amazonaws.com' # default: s3.amazonaws.com. - #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. - #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. - #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. - ``` + ```yaml + packages: + enabled: true + ## + ## The location where build packages are stored (default: shared/packages). + ## + #storage_path: shared/packages + object_store: + enabled: false + remote_directory: packages # The bucket name. + #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + #background_upload: true # Temporary option to limit automatic upload (Default: true). + #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + connection: + ## + ## If the provider is AWS S3, uncomment the following + ## + #provider: AWS + #region: us-east-1 + #aws_access_key_id: AWS_ACCESS_KEY_ID + #aws_secret_access_key: AWS_SECRET_ACCESS_KEY + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #host: 's3.amazonaws.com' # default: s3.amazonaws.com. + #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + ``` 1. [Restart GitLab] for the changes to take effect. diff --git a/doc/administration/pages/img/lets_encrypt_integration_v12_1.png b/doc/administration/pages/img/lets_encrypt_integration_v12_1.png Binary files differnew file mode 100644 index 00000000000..0f3ca25ce55 --- /dev/null +++ b/doc/administration/pages/img/lets_encrypt_integration_v12_1.png diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 3a7ca517d56..774e7056845 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -4,21 +4,24 @@ description: 'Learn how to administer GitLab Pages.' # GitLab Pages administration -> **Notes:** -> > - [Introduced][ee-80] in GitLab EE 8.3. > - Custom CNAMEs with TLS support were [introduced][ee-173] in GitLab EE 8.5. -> - GitLab Pages [were ported][ce-14605] to Community Edition in GitLab 8.17. -> - This guide is for Omnibus GitLab installations. If you have installed -> GitLab from source, follow the [Pages source installation document](source.md). -> - To learn how to use GitLab Pages, read the [user documentation][pages-userguide]. -> - Support for subgroup project's websites was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30548) in GitLab 11.8. - -This document describes how to set up the _latest_ GitLab Pages feature. Make -sure to read the [changelog](#changelog) if you are upgrading to a new GitLab +> - GitLab Pages [was ported][ce-14605] to Community Edition in GitLab 8.17. +> - Support for subgroup project's websites was +> [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30548) in GitLab 11.8. + +GitLab Pages allows for hosting of static sites. It must be configured by an +administrator. Separate [user documentation][pages-userguide] is available. + +Read the [changelog](#changelog) if you are upgrading to a new GitLab version as it may include new features and changes needed to be made in your configuration. +NOTE: **Note:** +This guide is for Omnibus GitLab installations. If you have installed +GitLab from source, see +[GitLab Pages administration for source installations](source.md). + ## Overview GitLab Pages makes use of the [GitLab Pages daemon], a simple HTTP server @@ -121,9 +124,9 @@ The Pages daemon doesn't listen to the outside world. 1. Set the external URL for GitLab Pages in `/etc/gitlab/gitlab.rb`: - ```shell - pages_external_url 'http://example.io' - ``` + ```shell + pages_external_url 'http://example.io' + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -146,16 +149,16 @@ outside world. 1. Place the certificate and key inside `/etc/gitlab/ssl` 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: - ```shell - pages_external_url 'https://example.io' + ```shell + pages_external_url 'https://example.io' - pages_nginx['redirect_http_to_https'] = true - pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" - pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/pages-nginx.key" - ``` + pages_nginx['redirect_http_to_https'] = true + pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" + pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/pages-nginx.key" + ``` - where `pages-nginx.crt` and `pages-nginx.key` are the SSL cert and key, - respectively. + where `pages-nginx.crt` and `pages-nginx.key` are the SSL cert and key, + respectively. 1. [Reconfigure GitLab][reconfigure]. @@ -168,9 +171,9 @@ behavior: 1. Edit `/etc/gitlab/gitlab.rb`. 1. Set the `inplace_chroot` to `true` for GitLab Pages: - ```shell - gitlab_pages['inplace_chroot'] = true - ``` + ```shell + gitlab_pages['inplace_chroot'] = true + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -203,16 +206,16 @@ world. Custom domains are supported, but no TLS. 1. Edit `/etc/gitlab/gitlab.rb`: - ```shell - pages_external_url "http://example.io" - nginx['listen_addresses'] = ['192.0.2.1'] - pages_nginx['enable'] = false - gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] - ``` + ```shell + pages_external_url "http://example.io" + nginx['listen_addresses'] = ['192.0.2.1'] + pages_nginx['enable'] = false + gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] + ``` - where `192.0.2.1` is the primary IP address that GitLab is listening to and - `192.0.2.2` and `2001::2` are the secondary IPs the GitLab Pages daemon - listens on. If you don't have IPv6, you can omit the IPv6 address. + where `192.0.2.1` is the primary IP address that GitLab is listening to and + `192.0.2.2` and `2001::2` are the secondary IPs the GitLab Pages daemon + listens on. If you don't have IPv6, you can omit the IPv6 address. 1. [Reconfigure GitLab][reconfigure]. @@ -234,26 +237,26 @@ world. Custom domains and TLS are supported. 1. Edit `/etc/gitlab/gitlab.rb`: - ```shell - pages_external_url "https://example.io" - nginx['listen_addresses'] = ['192.0.2.1'] - pages_nginx['enable'] = false - gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" - gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" - gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] - gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001::2]:443'] - ``` + ```shell + pages_external_url "https://example.io" + nginx['listen_addresses'] = ['192.0.2.1'] + pages_nginx['enable'] = false + gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" + gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" + gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] + gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001::2]:443'] + ``` - where `192.0.2.1` is the primary IP address that GitLab is listening to and - `192.0.2.2` and `2001::2` are the secondary IPs where the GitLab Pages daemon - listens on. If you don't have IPv6, you can omit the IPv6 address. + where `192.0.2.1` is the primary IP address that GitLab is listening to and + `192.0.2.2` and `2001::2` are the secondary IPs where the GitLab Pages daemon + listens on. If you don't have IPv6, you can omit the IPv6 address. 1. [Reconfigure GitLab][reconfigure]. ### Custom domain verification To prevent malicious users from hijacking domains that don't belong to them, -GitLab supports [custom domain verification](../../user/project/pages/getting_started_part_three.md#dns-txt-record). +GitLab supports [custom domain verification](../../user/project/pages/custom_domains_ssl_tls_certification/index.md#steps). When adding a custom domain, users will be required to prove they own it by adding a GitLab-controlled verification code to the DNS records for that domain. @@ -262,6 +265,23 @@ verification requirement. Navigate to `Admin area ➔ Settings` and uncheck **Require users to prove ownership of custom domains** in the Pages section. This setting is enabled by default. +### Let's Encrypt integration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28996) in GitLab 12.1. + +[GitLab Pages' Let's Encrypt integration](../../user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md) +allows users to add Let's Encrypt SSL certificates for GitLab Pages +sites served under a custom domain. + +To enable it, you'll need to: + +1. Choose an email on which you will recieve notifications about expiring domains. +1. Navigate to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. +1. Enter the email for receiving notifications and accept Let's Encrypt's Terms of Service as shown below. +1. Click **Save changes**. + + + ### Access control > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/33422) in GitLab 11.5. @@ -284,9 +304,9 @@ Pages access control is disabled by default. To enable it: 1. Enable it in `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_pages['access_control'] = true - ``` + ```ruby + gitlab_pages['access_control'] = true + ``` 1. [Reconfigure GitLab][reconfigure]. 1. Users can now configure it in their [projects' settings](../../user/project/pages/introduction.md#gitlab-pages-access-control-core-only). @@ -299,9 +319,9 @@ pages: 1. Configure in `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_pages['http_proxy'] = 'http://example:8080' - ``` + ```ruby + gitlab_pages['http_proxy'] = 'http://example:8080' + ``` 1. [Reconfigure Gitlab][reconfigure] for the changes to take effect. @@ -316,9 +336,9 @@ Follow the steps below to configure verbose logging of GitLab Pages daemon. If you wish to make it log events with level `DEBUG` you must configure this in `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_pages['log_verbose'] = true - ``` + ```shell + gitlab_pages['log_verbose'] = true + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -331,9 +351,9 @@ are stored. If you wish to store them in another location you must set it up in `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_rails['pages_path'] = "/mnt/storage/pages" - ``` + ```shell + gitlab_rails['pages_path'] = "/mnt/storage/pages" + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -344,19 +364,19 @@ Omnibus GitLab 11.1. 1. By default the listener is configured to listen for requests on `localhost:8090`. - If you wish to disable it you must configure this in - `/etc/gitlab/gitlab.rb`: + If you wish to disable it you must configure this in + `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_pages['listen_proxy'] = nil - ``` + ```shell + gitlab_pages['listen_proxy'] = nil + ``` - If you wish to make it listen on a different port you must configure this also in - `/etc/gitlab/gitlab.rb`: + If you wish to make it listen on a different port you must configure this also in + `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_pages['listen_proxy'] = "localhost:10080" - ``` + ```shell + gitlab_pages['listen_proxy'] = "localhost:10080" + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -378,28 +398,29 @@ Follow the steps below to configure GitLab Pages in a separate server. 1. On `app2` install GitLab omnibus and modify `/etc/gitlab/gitlab.rb` this way: - ```shell - external_url 'http://<ip-address-of-the-server>' - pages_external_url "http://<your-pages-domain>" - postgresql['enable'] = false - redis['enable'] = false - prometheus['enable'] = false - unicorn['enable'] = false - sidekiq['enable'] = false - gitlab_workhorse['enable'] = false - gitaly['enable'] = false - alertmanager['enable'] = false - node_exporter['enable'] = false - gitlab_rails['auto_migrate'] = false - ``` + ```shell + external_url 'http://<ip-address-of-the-server>' + pages_external_url "http://<your-pages-domain>" + postgresql['enable'] = false + redis['enable'] = false + prometheus['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + gitaly['enable'] = false + alertmanager['enable'] = false + node_exporter['enable'] = false + gitlab_rails['auto_migrate'] = false + ``` + 1. Run `sudo gitlab-ctl reconfigure`. 1. On `app1` apply the following changes to `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_pages['enable'] = false - pages_external_url "http://<your-pages-domain>" - gitlab_rails['pages_path'] = "/mnt/pages" - ``` + ```shell + gitlab_pages['enable'] = false + pages_external_url "http://<your-pages-domain>" + gitlab_rails['pages_path'] = "/mnt/pages" + ``` 1. Run `sudo gitlab-ctl reconfigure`. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 2100f7cd707..c77a1a9638f 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -24,8 +24,6 @@ SNI and exposes pages using HTTP2 by default. You are encouraged to read its [README][pages-readme] to fully understand how it works. ---- - In the case of [custom domains](#custom-domains) (but not [wildcard domains](#wildcard-domains)), the Pages daemon needs to listen on ports `80` and/or `443`. For that reason, there is some flexibility in the way @@ -92,8 +90,6 @@ since that is needed in all configurations. - [Wildcard DNS setup](#dns-configuration) ---- - URL scheme: `http://page.example.io` This is the minimum setup that you can use Pages with. It is the base for all @@ -102,90 +98,89 @@ The Pages daemon doesn't listen to the outside world. 1. Install the Pages daemon: - ``` - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git - cd gitlab-pages - sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) - sudo -u git -H make - ``` + ``` + cd /home/git + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git + cd gitlab-pages + sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) + sudo -u git -H make + ``` 1. Go to the GitLab installation directory: - ```bash - cd /home/git/gitlab - ``` + ```bash + cd /home/git/gitlab + ``` 1. Edit `gitlab.yml` and under the `pages` setting, set `enabled` to `true` and the `host` to the FQDN under which GitLab Pages will be served: - ```yaml - ## GitLab Pages - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - # path: shared/pages + ```yaml + ## GitLab Pages + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + # path: shared/pages - host: example.io - port: 80 - https: false - ``` + host: example.io + port: 80 + https: false + ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in order to enable the pages daemon. In `gitlab_pages_options` the `-pages-domain` must match the `host` setting that you set above. - ``` - gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090" - ``` + ``` + gitlab_pages_enabled=true + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090" + ``` 1. Copy the `gitlab-pages` Nginx configuration file: - ```bash - sudo cp lib/support/nginx/gitlab-pages /etc/nginx/sites-available/gitlab-pages.conf - sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf - ``` + ```bash + sudo cp lib/support/nginx/gitlab-pages /etc/nginx/sites-available/gitlab-pages.conf + sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf + ``` 1. Restart NGINX 1. [Restart GitLab][restart] ### Wildcard domains with TLS support -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Wildcard TLS certificate -> -> --- -> -> URL scheme: `https://page.example.io` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Wildcard TLS certificate + +URL scheme: `https://page.example.io` Nginx will proxy all requests to the daemon. Pages daemon doesn't listen to the outside world. 1. Install the Pages daemon: - ``` - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git - cd gitlab-pages - sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) - sudo -u git -H make - ``` + ``` + cd /home/git + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git + cd gitlab-pages + sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) + sudo -u git -H make + ``` 1. In `gitlab.yml`, set the port to `443` and https to `true`: - ```bash - ## GitLab Pages - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - # path: shared/pages + ```bash + ## GitLab Pages + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + # path: shared/pages - host: example.io - port: 443 - https: true - ``` + host: example.io + port: 443 + https: true + ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in order to enable the pages daemon. In `gitlab_pages_options` the @@ -193,17 +188,17 @@ outside world. The `-root-cert` and `-root-key` settings are the wildcard TLS certificates of the `example.io` domain: - ``` - gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key - ``` + ``` + gitlab_pages_enabled=true + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key + ``` 1. Copy the `gitlab-pages-ssl` Nginx configuration file: - ```bash - sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf - sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf - ``` + ```bash + sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf + sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf + ``` 1. Restart NGINX 1. [Restart GitLab][restart] @@ -217,13 +212,12 @@ that without TLS certificates. ### Custom domains -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Secondary IP -> -> --- -> -> URL scheme: `http://page.example.io` and `http://domain.com` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Secondary IP + +URL scheme: `http://page.example.io` and `http://domain.com` In that case, the pages daemon is running, Nginx still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -231,48 +225,48 @@ world. Custom domains are supported, but no TLS. 1. Install the Pages daemon: - ``` - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git - cd gitlab-pages - sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) - sudo -u git -H make - ``` + ``` + cd /home/git + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git + cd gitlab-pages + sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) + sudo -u git -H make + ``` 1. Edit `gitlab.yml` to look like the example below. You need to change the `host` to the FQDN under which GitLab Pages will be served. Set `external_http` to the secondary IP on which the pages daemon will listen for connections: - ```yaml - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - # path: shared/pages + ```yaml + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + # path: shared/pages - host: example.io - port: 80 - https: false + host: example.io + port: 80 + https: false - external_http: 192.0.2.2:80 - ``` + external_http: 192.0.2.2:80 + ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in order to enable the pages daemon. In `gitlab_pages_options` the `-pages-domain` and `-listen-http` must match the `host` and `external_http` settings that you set above respectively: - ``` - gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 192.0.2.2:80" - ``` + ``` + gitlab_pages_enabled=true + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 192.0.2.2:80" + ``` 1. Copy the `gitlab-pages-ssl` Nginx configuration file: - ```bash - sudo cp lib/support/nginx/gitlab-pages /etc/nginx/sites-available/gitlab-pages.conf - sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf - ``` + ```bash + sudo cp lib/support/nginx/gitlab-pages /etc/nginx/sites-available/gitlab-pages.conf + sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf + ``` 1. Edit all GitLab related configs in `/etc/nginx/site-available/` and replace `0.0.0.0` with `192.0.2.1`, where `192.0.2.1` the primary IP where GitLab @@ -282,14 +276,13 @@ world. Custom domains are supported, but no TLS. ### Custom domains with TLS support -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Wildcard TLS certificate -> - Secondary IP -> -> --- -> -> URL scheme: `https://page.example.io` and `https://domain.com` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Wildcard TLS certificate +- Secondary IP + +URL scheme: `https://page.example.io` and `https://domain.com` In that case, the pages daemon is running, Nginx still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -297,33 +290,33 @@ world. Custom domains and TLS are supported. 1. Install the Pages daemon: - ``` - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git - cd gitlab-pages - sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) - sudo -u git -H make - ``` + ``` + cd /home/git + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git + cd gitlab-pages + sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) + sudo -u git -H make + ``` 1. Edit `gitlab.yml` to look like the example below. You need to change the `host` to the FQDN under which GitLab Pages will be served. Set `external_http` and `external_https` to the secondary IP on which the pages daemon will listen for connections: - ```yaml - ## GitLab Pages - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - # path: shared/pages + ```yaml + ## GitLab Pages + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + # path: shared/pages - host: example.io - port: 443 - https: true + host: example.io + port: 443 + https: true - external_http: 192.0.2.2:80 - external_https: 192.0.2.2:443 - ``` + external_http: 192.0.2.2:80 + external_https: 192.0.2.2:443 + ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in order to enable the pages daemon. In `gitlab_pages_options` the @@ -332,17 +325,17 @@ world. Custom domains and TLS are supported. The `-root-cert` and `-root-key` settings are the wildcard TLS certificates of the `example.io` domain: - ``` - gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 192.0.2.2:80 -listen-https 192.0.2.2:443 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key - ``` + ``` + gitlab_pages_enabled=true + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 192.0.2.2:80 -listen-https 192.0.2.2:443 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key + ``` 1. Copy the `gitlab-pages-ssl` Nginx configuration file: - ```bash - sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf - sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf - ``` + ```bash + sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf + sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf + ``` 1. Edit all GitLab related configs in `/etc/nginx/site-available/` and replace `0.0.0.0` with `192.0.2.1`, where `192.0.2.1` the primary IP where GitLab @@ -359,9 +352,9 @@ are stored. If you wish to store them in another location you must set it up in `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['pages_path'] = "/mnt/storage/pages" - ``` + ```ruby + gitlab_rails['pages_path'] = "/mnt/storage/pages" + ``` 1. [Reconfigure GitLab][reconfigure] @@ -414,10 +407,10 @@ Pages access control is disabled by default. To enable it: 1. Modify your `config/gitlab.yml` file: - ```yaml - pages: - access_control: true - ``` + ```yaml + pages: + access_control: true + ``` 1. [Restart GitLab][restart]. 1. Create a new [system OAuth application](../../integration/oauth_provider.md#adding-an-application-through-the-profile). @@ -426,12 +419,12 @@ Pages access control is disabled by default. To enable it: application, but it does need the "api" scope. 1. Start the Pages daemon with the following additional arguments: - ```shell - -auth-client-secret <OAuth code generated by GitLab> \ - -auth-redirect-uri http://projects.example.io/auth \ - -auth-secret <40 random hex characters> \ - -auth-server <URL of the GitLab instance> - ``` + ```shell + -auth-client-secret <OAuth code generated by GitLab> \ + -auth-redirect-uri http://projects.example.io/auth \ + -auth-secret <40 random hex characters> \ + -auth-server <URL of the GitLab instance> + ``` 1. Users can now configure it in their [projects' settings](../../user/project/pages/introduction.md#gitlab-pages-access-control-core-only). @@ -444,12 +437,12 @@ are stored. If you wish to store them in another location you must set it up in `gitlab.yml` under the `pages` section: - ```yaml - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - path: /mnt/storage/pages - ``` + ```yaml + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + path: /mnt/storage/pages + ``` 1. [Restart GitLab][restart] diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index 4302667caf5..4cf3c607dae 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -52,7 +52,37 @@ as appropriate. The plugins file list is updated for each event, there is no need to restart GitLab to apply a new plugin. If a plugin executes with non-zero exit code or GitLab fails to execute it, a -message will be logged to `plugin.log`. +message will be logged to: + +- `gitlab-rails/plugin.log` in an Omnibus installation. +- `log/plugin.log` in a source installation. + +## Creating plugins + +Below is an example that will only response on the event `project_create` and +will inform the admins from the GitLab instance that a new project has been created. + +```ruby +# By using the embedded ruby version we eliminate the possibility that our chosen language +# would be unavailable from +#!/opt/gitlab/embedded/bin/ruby +require 'json' +require 'mail' + +# The incoming variables are in JSON format so we need to parse it first. +ARGS = JSON.parse(STDIN.read) + +# We only want to trigger this plugin on the event project_create +return unless ARGS['event_name'] == 'project_create' + +# We will inform our admins of our gitlab instance that a new project is created +Mail.deliver do + from 'info@gitlab_instance.com' + to 'admin@gitlab_instance.com' + subject "new project " + ARGS['name'] + body ARGS['owner_name'] + 'created project ' + ARGS['name'] +end +``` ## Validation diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 036e1d3fe61..716a4259a64 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -1,4 +1,4 @@ -# Pseudonymizer **[ULTIMATE]** +# Pseudonymizer **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5532) in [GitLab Ultimate][ee] 11.1. @@ -22,36 +22,36 @@ To configure the pseudonymizer, you need to: - Provide a manifest file that describes which fields should be included or pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/config/pseudonymizer.yml)). - A default manifest is provided with the GitLab installation. Using a relative file path will be resolved from the Rails root. + A default manifest is provided with the GitLab installation. Using a relative file path will be resolved from the Rails root. Alternatively, you can use an absolute file path. -- Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. +- Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. **For Omnibus installations:** 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: - ```ruby - gitlab_rails['pseudonymizer_manifest'] = 'config/pseudonymizer.yml' - gitlab_rails['pseudonymizer_upload_remote_directory'] = 'gitlab-elt' # bucket name - gitlab_rails['pseudonymizer_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', - 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' - } - ``` - - NOTE: **Note:** - If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. - - ```ruby - gitlab_rails['pseudonymizer_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'use_iam_profile' => true - } - ``` + ```ruby + gitlab_rails['pseudonymizer_manifest'] = 'config/pseudonymizer.yml' + gitlab_rails['pseudonymizer_upload_remote_directory'] = 'gitlab-elt' # bucket name + gitlab_rails['pseudonymizer_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: **Note:** + If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. + + ```ruby + gitlab_rails['pseudonymizer_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -63,17 +63,17 @@ To configure the pseudonymizer, you need to: 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - pseudonymizer: - manifest: config/pseudonymizer.yml - upload: - remote_directory: 'gitlab-elt' # bucket name - connection: - provider: AWS - aws_access_key_id: AWS_ACCESS_KEY_ID - aws_secret_access_key: AWS_SECRET_ACCESS_KEY - region: eu-central-1 - ``` + ```yaml + pseudonymizer: + manifest: config/pseudonymizer.yml + upload: + remote_directory: 'gitlab-elt' # bucket name + connection: + provider: AWS + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 9f3b31442f3..691d34ab7fa 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -1,8 +1,8 @@ -# Geo Rake Tasks **[PREMIUM ONLY]** +# Geo Rake Tasks **(PREMIUM ONLY)** ## Git housekeeping -There are few tasks you can run to schedule a git housekeeping to start at the +There are few tasks you can run to schedule a git housekeeping to start at the next repository sync in a **Secondary node**: ### Incremental Repack @@ -23,7 +23,7 @@ sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_EN ### Full Repack -This is equivalent of running `git repack -d -A --pack-kept-objects` on a +This is equivalent of running `git repack -d -A --pack-kept-objects` on a _bare_ repository which will optionally, write a reachability bitmap index when this is enabled in GitLab. diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 91fc0133d56..e880d76e756 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -28,6 +28,31 @@ limit by passing a number to the check task: rake gitlab:ldap:check[50] ``` +## Run a Group Sync + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.3. + +The following task will run a [group sync](../auth/ldap-ee.md#group-sync) immediately. This is valuable +when you'd like to update all configured group memberships against LDAP without +waiting for the next scheduled group sync to be run. + +NOTE: **NOTE:** +If you'd like to change the frequency at which a group sync is performed, +[adjust the cron schedule](../auth/ldap-ee.md#adjusting-ldap-group-sync-schedule) +instead. + +**Omnibus Installation** + +``` +sudo gitlab-rake gitlab:ldap:group_sync +``` + +**Source Installation** + +```bash +bundle exec rake gitlab:ldap:group_sync +``` + ## Rename a provider If you change the LDAP server ID in `gitlab.yml` or `gitlab.rb` you will need diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 0b4c1ae15d6..8d0b5b42515 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -61,8 +61,8 @@ It will check that each component was set up according to the installation guide You may also have a look at our Troubleshooting Guides: -- [Troubleshooting Guide (GitLab)](http://docs.gitlab.com/ee/README.html#troubleshooting) -- [Troubleshooting Guide (Omnibus Gitlab)](http://docs.gitlab.com/omnibus/README.html#troubleshooting) +- [Troubleshooting Guide (GitLab)](../index.md#troubleshooting) +- [Troubleshooting Guide (Omnibus Gitlab)](https://docs.gitlab.com/omnibus/README.html#troubleshooting) **Omnibus Installation** @@ -251,3 +251,22 @@ sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:*] # to clear a lease for repository garbage collection in a specific project: (id=4) sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:4] ``` + +## Display status of database migrations + +To check the status of migrations, you can use the following rake task: + +```bash +sudo gitlab-rake db:migrate:status +``` + +This will output a table with a `Status` of `up` or `down` for +each Migration ID. + +```bash +database: gitlabhq_production + + Status Migration ID Migration Name +-------------------------------------------------- + up migration_id migration_name +```
\ No newline at end of file diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 6ca23aabdec..138db4dfbb1 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -1,15 +1,15 @@ -# Project import/export administration **[CORE ONLY]** +# Project import/export administration **(CORE ONLY)** >**Note:** > -> - [Introduced][ce-3050] in GitLab 8.9. -> - Importing will not be possible if the import instance version is lower -> than that of the exporter. -> - For existing installations, the project import option has to be enabled in -> application settings (`/admin/application_settings`) under 'Import sources'. -> - The exports are stored in a temporary [shared directory][tmp] and are deleted -> every 24 hours by a specific worker. -> - ImportExport can use object storage automatically starting from GitLab 11.3 +> - [Introduced][ce-3050] in GitLab 8.9. +> - Importing will not be possible if the import instance version is lower +> than that of the exporter. +> - For existing installations, the project import option has to be enabled in +> application settings (`/admin/application_settings`) under 'Import sources'. +> - The exports are stored in a temporary [shared directory][tmp] and are deleted +> every 24 hours by a specific worker. +> - ImportExport can use object storage automatically starting from GitLab 11.3 The GitLab Import/Export version can be checked by using: diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 42a1a1c2e60..1198f3414c5 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,17 +1,17 @@ # Repository Storage Rake Tasks -This is a collection of rake tasks you can use to help you list and migrate -existing projects and attachments associated with it from Legacy storage to +This is a collection of rake tasks you can use to help you list and migrate +existing projects and attachments associated with it from Legacy storage to the new Hashed storage type. You can read more about the storage types [here][storage-types]. ## Migrate existing projects to Hashed storage -Before migrating your existing projects, you should +Before migrating your existing projects, you should [enable hashed storage][storage-migration] for the new projects as well. -This task will schedule all your existing projects and attachments associated with it to be migrated to the +This task will schedule all your existing projects and attachments associated with it to be migrated to the **Hashed** storage type: **Omnibus Installation** @@ -30,15 +30,15 @@ They both also accept a range as environment variable: ```bash # to migrate any non migrated project from ID 20 to 50. -export ID_FROM=20 +export ID_FROM=20 export ID_TO=50 ``` You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. -There is a specific Queue you can watch to see how long it will take to finish: +There is a specific Queue you can watch to see how long it will take to finish: `hashed_storage:hashed_storage_project_migrate` -After it reaches zero, you can confirm every project has been migrated by running the commands bellow. +After it reaches zero, you can confirm every project has been migrated by running the commands bellow. If you find it necessary, you can run this migration script again to schedule missing projects. Any error or warning will be logged in Sidekiq's log file. @@ -55,7 +55,7 @@ If you need to rollback the storage migration for any reason, you can follow the NOTE: **Note:** Hashed Storage will be required in future version of GitLab. -To prevent new projects from being created in the Hashed storage, +To prevent new projects from being created in the Hashed storage, you need to undo the [enable hashed storage][storage-migration] changes. This task will schedule all your existing projects and associated attachments to be rolled back to the @@ -77,15 +77,14 @@ Both commands accept a range as environment variable: ```bash # to rollback any migrated project from ID 20 to 50. -export ID_FROM=20 +export ID_FROM=20 export ID_TO=50 ``` You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. - -After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. +After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. Any error or warning will be logged in Sidekiq's log file. @@ -106,8 +105,6 @@ sudo gitlab-rake gitlab:storage:legacy_projects sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production ``` ------- - To list projects using **Legacy** storage: **Omnibus Installation** @@ -139,8 +136,6 @@ sudo gitlab-rake gitlab:storage:hashed_projects sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production ``` ------- - To list projects using **Hashed** storage: **Omnibus Installation** @@ -171,8 +166,6 @@ sudo gitlab-rake gitlab:storage:legacy_attachments sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production ``` ------- - To list project attachments using **Legacy** storage: **Omnibus Installation** @@ -203,8 +196,6 @@ sudo gitlab-rake gitlab:storage:hashed_attachments sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production ``` ------- - To list project attachments using **Hashed** storage: **Omnibus Installation** diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index fd8ea8d3162..86e8b763f51 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -103,3 +103,13 @@ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, S sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" ``` + +## Migrate legacy uploads out of deprecated paths + +> Introduced in GitLab 12.3. + +To migrate all uploads created by legacy uploaders, run: + +```shell +bundle exec rake gitlab:uploads:legacy:migrate +``` diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 54a423b9571..ae5ccfb9e37 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -4,16 +4,16 @@ You need `exiftool` installed on your system. If you installed GitLab: -- Using the Omnibus package, you're all set. -- From source, make sure `exiftool` is installed: +- Using the Omnibus package, you're all set. +- From source, make sure `exiftool` is installed: - ```sh - # Debian/Ubuntu - sudo apt-get install libimage-exiftool-perl + ```sh + # Debian/Ubuntu + sudo apt-get install libimage-exiftool-perl - # RHEL/CentOS - sudo yum install perl-Image-ExifTool - ``` + # RHEL/CentOS + sudo yum install perl-Image-ExifTool + ``` ## Remove EXIF data from existing uploads diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index d57fc67c83e..406f7e8a034 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -14,109 +14,109 @@ The instructions make the assumption that you will be using the email address `i 1. Install the `postfix` package if it is not installed already: - ```sh - sudo apt-get install postfix - ``` + ```sh + sudo apt-get install postfix + ``` - When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches `gitlab.example.com`. + When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches gitlab.example.com`. 1. Install the `mailutils` package. - ```sh - sudo apt-get install mailutils - ``` + ```sh + sudo apt-get install mailutils + ``` ## Create user 1. Create a user for incoming email. - ```sh - sudo useradd -m -s /bin/bash incoming - ``` + ```sh + sudo useradd -m -s /bin/bash incoming + ``` 1. Set a password for this user. - ```sh - sudo passwd incoming - ``` + ```sh + sudo passwd incoming + ``` - Be sure not to forget this, you'll need it later. + Be sure not to forget this, you'll need it later. ## Test the out-of-the-box setup 1. Connect to the local SMTP server: - ```sh - telnet localhost 25 - ``` + ```sh + telnet localhost 25 + ``` - You should see a prompt like this: + You should see a prompt like this: - ```sh - Trying 127.0.0.1... - Connected to localhost. - Escape character is '^]'. - 220 gitlab.example.com ESMTP Postfix (Ubuntu) - ``` + ```sh + Trying 127.0.0.1... + Connected to localhost. + Escape character is '^]'. + 220 gitlab.example.com ESMTP Postfix (Ubuntu) + ``` - If you get a `Connection refused` error instead, verify that `postfix` is running: + If you get a `Connection refused` error instead, verify that `postfix` is running: - ```sh - sudo postfix status - ``` + ```sh + sudo postfix status + ``` - If it is not, start it: + If it is not, start it: - ```sh - sudo postfix start - ``` + ```sh + sudo postfix start + ``` 1. Send the new `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: - ``` - ehlo localhost - mail from: root@localhost - rcpt to: incoming@localhost - data - Subject: Re: Some issue + ``` + ehlo localhost + mail from: root@localhost + rcpt to: incoming@localhost + data + Subject: Re: Some issue - Sounds good! - . - quit - ``` + Sounds good! + . + quit + ``` - _**Note:** The `.` is a literal period on its own line._ + _**Note:** The `.` is a literal period on its own line._ - _**Note:** If you receive an error after entering `rcpt to: incoming@localhost` - then your Postfix `my_network` configuration is not correct. The error will - say 'Temporary lookup failure'. See - [Configure Postfix to receive email from the Internet](#configure-postfix-to-receive-email-from-the-internet)._ + _**Note:** If you receive an error after entering `rcpt to: incoming@localhost` + then your Postfix `my_network` configuration is not correct. The error will + say 'Temporary lookup failure'. See + [Configure Postfix to receive email from the Internet](#configure-postfix-to-receive-email-from-the-internet)._ 1. Check if the `incoming` user received the email: - ```sh - su - incoming - mail - ``` + ```sh + su - incoming + mail + ``` - You should see output like this: + You should see output like this: - ``` - "/var/mail/incoming": 1 message 1 unread - >U 1 root@localhost 59/2842 Re: Some issue - ``` + ``` + "/var/mail/incoming": 1 message 1 unread + >U 1 root@localhost 59/2842 Re: Some issue + ``` - Quit the mail app: + Quit the mail app: - ```sh - q - ``` + ```sh + q + ``` 1. Log out of the `incoming` account and go back to being `root`: - ```sh - logout - ``` + ```sh + logout + ``` ## Configure Postfix to use Maildir-style mailboxes @@ -124,208 +124,212 @@ Courier, which we will install later to add IMAP authentication, requires mailbo 1. Configure Postfix to use Maildir-style mailboxes: - ```sh - sudo postconf -e "home_mailbox = Maildir/" - ``` + ```sh + sudo postconf -e "home_mailbox = Maildir/" + ``` 1. Restart Postfix: - ```sh - sudo /etc/init.d/postfix restart - ``` + ```sh + sudo /etc/init.d/postfix restart + ``` 1. Test the new setup: - 1. Follow steps 1 and 2 of _[Test the out-of-the-box setup](#test-the-out-of-the-box-setup)_. - 1. Check if the `incoming` user received the email: + 1. Follow steps 1 and 2 of _[Test the out-of-the-box setup](#test-the-out-of-the-box-setup)_. + 1. Check if the `incoming` user received the email: - ```sh - su - incoming - MAIL=/home/incoming/Maildir - mail - ``` + ```sh + su - incoming + MAIL=/home/incoming/Maildir + mail + ``` - You should see output like this: + You should see output like this: - ``` - "/home/incoming/Maildir": 1 message 1 unread - >U 1 root@localhost 59/2842 Re: Some issue - ``` + ``` + "/home/incoming/Maildir": 1 message 1 unread + >U 1 root@localhost 59/2842 Re: Some issue + ``` - Quit the mail app: + Quit the mail app: - ```sh - q - ``` + ```sh + q + ``` - _**Note:** If `mail` returns an error `Maildir: Is a directory` then your - version of `mail` doesn't support Maildir style mailboxes. Install - `heirloom-mailx` by running `sudo apt-get install heirloom-mailx`. Then, - try the above steps again, substituting `heirloom-mailx` for the `mail` - command._ + _**Note:** If `mail` returns an error `Maildir: Is a directory` then your + version of `mail` doesn't support Maildir style mailboxes. Install + `heirloom-mailx` by running `sudo apt-get install heirloom-mailx`. Then, + try the above steps again, substituting `heirloom-mailx` for the `mail` + command._ 1. Log out of the `incoming` account and go back to being `root`: - ```sh - logout - ``` + ```sh + logout + ``` ## Install the Courier IMAP server 1. Install the `courier-imap` package: - ```sh - sudo apt-get install courier-imap - ``` + ```sh + sudo apt-get install courier-imap + ``` - And start `imapd`: - ```sh - imapd start - ``` + And start `imapd`: + + ```sh + imapd start + ``` 1. The courier-authdaemon isn't started after installation. Without it, imap authentication will fail: - ```sh - sudo service courier-authdaemon start - ``` - You can also configure courier-authdaemon to start on boot: - ```sh - sudo systemctl enable courier-authdaemon - ``` + + ```sh + sudo service courier-authdaemon start + ``` + + You can also configure courier-authdaemon to start on boot: + + ```sh + sudo systemctl enable courier-authdaemon + ``` ## Configure Postfix to receive email from the internet 1. Let Postfix know about the domains that it should consider local: - ```sh - sudo postconf -e "mydestination = gitlab.example.com, localhost.localdomain, localhost" - ``` + ```sh + sudo postconf -e "mydestination = gitlab.example.com, localhost.localdomain, localhost" + ``` 1. Let Postfix know about the IPs that it should consider part of the LAN: - We'll assume `192.168.1.0/24` is your local LAN. You can safely skip this step if you don't have other machines in the same local network. + We'll assume `192.168.1.0/24` is your local LAN. You can safely skip this step if you don't have other machines in the same local network. - ```sh - sudo postconf -e "mynetworks = 127.0.0.0/8, 192.168.1.0/24" - ``` + ```sh + sudo postconf -e "mynetworks = 127.0.0.0/8, 192.168.1.0/24" + ``` 1. Configure Postfix to receive mail on all interfaces, which includes the internet: - ```sh - sudo postconf -e "inet_interfaces = all" - ``` + ```sh + sudo postconf -e "inet_interfaces = all" + ``` 1. Configure Postfix to use the `+` delimiter for sub-addressing: - ```sh - sudo postconf -e "recipient_delimiter = +" - ``` + ```sh + sudo postconf -e "recipient_delimiter = +" + ``` 1. Restart Postfix: - ```sh - sudo service postfix restart - ``` + ```sh + sudo service postfix restart + ``` ## Test the final setup 1. Test SMTP under the new setup: - 1. Connect to the SMTP server: + 1. Connect to the SMTP server: - ```sh - telnet gitlab.example.com 25 - ``` + ```sh + telnet gitlab.example.com 25 + ``` - You should see a prompt like this: + You should see a prompt like this: - ```sh - Trying 123.123.123.123... - Connected to gitlab.example.com. - Escape character is '^]'. - 220 gitlab.example.com ESMTP Postfix (Ubuntu) - ``` + ```sh + Trying 123.123.123.123... + Connected to gitlab.example.com. + Escape character is '^]'. + 220 gitlab.example.com ESMTP Postfix (Ubuntu) + ``` - If you get a `Connection refused` error instead, make sure your firewall is set up to allow inbound traffic on port 25. + If you get a `Connection refused` error instead, make sure your firewall is set up to allow inbound traffic on port 25. - 1. Send the `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + 1. Send the `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: - ``` - ehlo gitlab.example.com - mail from: root@gitlab.example.com - rcpt to: incoming@gitlab.example.com - data - Subject: Re: Some issue + ``` + ehlo gitlab.example.com + mail from: root@gitlab.example.com + rcpt to: incoming@gitlab.example.com + data + Subject: Re: Some issue - Sounds good! - . - quit - ``` + Sounds good! + . + quit + ``` - (Note: The `.` is a literal period on its own line) + (Note: The `.` is a literal period on its own line) - 1. Check if the `incoming` user received the email: + 1. Check if the `incoming` user received the email: - ```sh - su - incoming - MAIL=/home/incoming/Maildir - mail - ``` + ```sh + su - incoming + MAIL=/home/incoming/Maildir + mail + ``` - You should see output like this: + You should see output like this: - ``` - "/home/incoming/Maildir": 1 message 1 unread - >U 1 root@gitlab.example.com 59/2842 Re: Some issue - ``` + ``` + "/home/incoming/Maildir": 1 message 1 unread + >U 1 root@gitlab.example.com 59/2842 Re: Some issue + ``` - Quit the mail app: + Quit the mail app: - ```sh - q - ``` + ```sh + q + ``` - 1. Log out of the `incoming` account and go back to being `root`: + 1. Log out of the `incoming` account and go back to being `root`: - ```sh - logout - ``` + ```sh + logout + ``` 1. Test IMAP under the new setup: - 1. Connect to the IMAP server: + 1. Connect to the IMAP server: - ```sh - telnet gitlab.example.com 143 - ``` + ```sh + telnet gitlab.example.com 143 + ``` - You should see a prompt like this: + You should see a prompt like this: - ```sh - Trying 123.123.123.123... - Connected to mail.example.gitlab.com. - Escape character is '^]'. - - OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc. See COPYING for distribution information. - ``` + ```sh + Trying 123.123.123.123... + Connected to mail.example.gitlab.com. + Escape character is '^]'. + - OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc. See COPYING for distribution information. + ``` - 1. Sign in as the `incoming` user to test IMAP, by entering the following into the IMAP prompt: + 1. Sign in as the `incoming` user to test IMAP, by entering the following into the IMAP prompt: - ``` - a login incoming PASSWORD - ``` + ``` + a login incoming PASSWORD + ``` - Replace PASSWORD with the password you set on the `incoming` user earlier. + Replace PASSWORD with the password you set on the `incoming` user earlier. - You should see output like this: + You should see output like this: - ``` - a OK LOGIN Ok. - ``` + ``` + a OK LOGIN Ok. + ``` - 1. Disconnect from the IMAP server: + 1. Disconnect from the IMAP server: - ```sh - a logout - ``` + ```sh + a logout + ``` ## Done! diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 7cf8f20a9dc..05a7cb006a3 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -33,8 +33,8 @@ in `repocheck.log`: - in the [admin panel](logs.md#repochecklog) - or on disk, see: - - `/var/log/gitlab/gitlab-rails` for Omnibus installations - - `/home/git/gitlab/log` for installations from source + - `/var/log/gitlab/gitlab-rails` for Omnibus installations + - `/home/git/gitlab/log` for installations from source If for some reason the periodic repository check caused a lot of false alarms you can choose to clear *all* repository check states by diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 4aafc06cfdc..b1a870210a8 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -55,31 +55,29 @@ storage2: > don't specify the full repository path but the parent path), but not for source > installations. ---- - Now that you've read that big fat warning above, let's edit the configuration files and add the full paths of the alternative repository storage paths. In -the example below, we add two more mountpoints that are named `nfs` and `cephfs` +the example below, we add two more mountpoints that are named `nfs_1` and `nfs_2` respectively. -NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. +NOTE: **Note:** This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. **For installations from source** 1. Edit `gitlab.yml` and add the storage paths: - ```yaml - repositories: - # Paths where repositories can be stored. Give the canonicalized absolute pathname. - # NOTE: REPOS PATHS MUST NOT CONTAIN ANY SYMLINK!!! - storages: # You must have at least a 'default' storage path. - default: - path: /home/git/repositories - nfs: - path: /mnt/nfs/repositories - cephfs: - path: /mnt/cephfs/repositories - ``` + ```yaml + repositories: + # Paths where repositories can be stored. Give the canonicalized absolute pathname. + # NOTE: REPOS PATHS MUST NOT CONTAIN ANY SYMLINK!!! + storages: # You must have at least a 'default' storage path. + default: + path: /home/git/repositories + nfs_1: + path: /mnt/nfs1/repositories + nfs_2: + path: /mnt/nfs2/repositories + ``` 1. [Restart GitLab][restart-gitlab] for the changes to take effect. @@ -90,23 +88,21 @@ are upgrading from a version prior to 8.10, make sure to add the configuration as described in the step above. After you make the changes and confirm they are working, you can remove the `repos_path` line. ---- - **For Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the default one: - ```ruby - git_data_dirs({ - "default" => { "path" => "/var/opt/gitlab/git-data" }, - "nfs" => { "path" => "/mnt/nfs/git-data" }, - "cephfs" => { "path" => "/mnt/cephfs/git-data" } - }) - ``` + ```ruby + git_data_dirs({ + "default" => { "path" => "/var/opt/gitlab/git-data" }, + "nfs_1" => { "path" => "/mnt/nfs1/git-data" }, + "nfs_2" => { "path" => "/mnt/nfs2/git-data" } + }) + ``` - Note that Omnibus stores the repositories in a `repositories` subdirectory - of the `git-data` directory. + Note that Omnibus stores the repositories in a `repositories` subdirectory + of the `git-data` directory. ## Choose where new project repositories will be stored diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 38842693d73..1669f8b128c 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -6,19 +6,19 @@ Two different storage layouts can be used to store the repositories on disk and their characteristics. GitLab can be configured to use one or multiple repository shard locations -that can be: +that can be: - Mounted to the local disk - Exposed as an NFS shared volume - Acessed via [gitaly] on its own machine. In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` -configuration hash. The storage layouts discussed here will apply to any shard +configuration hash. The storage layouts discussed here will apply to any shard defined in it. The `default` repository shard that is available in any installations that haven't customized it, points to the local folder: `/var/opt/gitlab/git-data`. -Anything discussed below is expected to be part of that folder. +Anything discussed below is expected to be part of that folder. ## Legacy Storage @@ -47,18 +47,12 @@ Any change in the URL will need to be reflected on disk (when groups / users or projects are renamed). This can add a lot of load in big installations, especially if using any type of network based filesystem. -CAUTION: **Caution:** -For Geo in particular: Geo does work with legacy storage, but in some -edge cases due to race conditions it can lead to errors when a project is -renamed multiple times in short succession, or a project is deleted and -recreated under the same name very quickly. We expect these race events to be -rare, and we have not observed a race condition side-effect happening yet. -This pattern also exists in other objects stored in GitLab, like issue -Attachments, GitLab Pages artifacts, Docker Containers for the integrated -Registry, etc. Hashed storage is a requirement for Geo. - ## Hashed Storage +CAUTION: **Important:** +Geo requires Hashed Storage since 12.0. If you haven't migrated yet, +check the [migration instructions](#how-to-migrate-to-hashed-storage) ASAP. + Hashed Storage is the new storage behavior we rolled out with 10.0. Instead of coupling project URL and the folder structure where the repository will be stored on disk, we are coupling a hash, based on the project's ID. This makes @@ -86,25 +80,20 @@ by another folder with the next 2 characters. They are both stored in a special ### Hashed object pools -CAUTION: **Beta:** -Hashed objects pools are considered beta, and are not ready for production use. -Follow [gitaly#1548](https://gitlab.com/gitlab-org/gitaly/issues/1548) for -updates. +> [Introduced](https://gitlab.com/gitlab-org/gitaly/issues/1606) in GitLab 12.1. -For deduplication of public forks and their parent repository, objects are pooled -in an object pool. These object pools are a third repository where shared objects -are stored. +Forks of public projects are deduplicated by creating a third repository, the object pool, containing the objects from the source project. Using `objects/info/alternates`, the source project and forks use the object pool for shared objects. Objects are moved from the source project to the object pool when housekeeping is run on the source project. ```ruby # object pool paths "@pools/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" ``` -The object pool feature is behind the `object_pools` feature flag, and can be -enabled for individual projects by executing -`Feature.enable(:object_pools, Project.find(<id>))`. Note that the project has to -be on hashed storage, should not be a fork itself, and hashed storage should be -enabled for all new projects. +Object pools can be disabled using the `object_pools` feature flag, and can be +disabled for individual projects by executing +`Feature.disable(:object_pools, Project.find(<id>))`. Disabling object pools +will not change existing deduplicated forks, but will prevent new forks from +being deduplicated. DANGER: **Danger:** Do not run `git prune` or `git gc` in pool repositories! This can @@ -114,9 +103,9 @@ question. ### How to migrate to Hashed Storage To start a migration, enable Hashed Storage for new projects: - + 1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. -2. Select the **Use hashed storage paths for newly created and renamed projects** checkbox. +1. Select the **Use hashed storage paths for newly created and renamed projects** checkbox. Check if the change breaks any existing integration you may have that either runs on the same machine as your repositories are located, or may login to that machine @@ -135,7 +124,7 @@ an Omnibus Gitlab installation: sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 ``` -Check the [documentation][rake/migrate-to-hashed] for additional information and instructions for +Check the [documentation][rake/migrate-to-hashed] for additional information and instructions for source-based installation. #### Rollback @@ -144,14 +133,14 @@ Similar to the migration, to disable Hashed Storage for new projects: 1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. -2. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. +1. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. -To schedule a complete rollback, see the +To schedule a complete rollback, see the [rake task documentation for storage rollback](raketasks/storage.md#rollback-from-hashed-storage-to-legacy-storage) for instructions. The rollback task also supports specifying a range of Project IDs. Here is an example of limiting the rollout to Project IDs 50 to 100, in an Omnibus Gitlab installation: - + ```bash sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 ``` diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index cbc3fbd9473..e23f2052d04 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -137,9 +137,9 @@ If you are using other init systems, like systemd, you can check the [GitLab Recipes][gl-recipes] repository for some unofficial services. These are **not** officially supported so use them at your own risk. -[omnibus-dl]: https://about.gitlab.com/downloads/ "Download the Omnibus packages" +[omnibus-dl]: https://about.gitlab.com/install/ "Download the Omnibus packages" [install]: ../install/installation.md "Documentation to install GitLab from source" [mailroom]: reply_by_email.md "Used for replying by email in GitLab issues and merge requests" -[chef]: https://www.chef.io/chef/ "Chef official website" +[chef]: https://www.chef.io/products/chef-infra/ "Chef official website" [src-service]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/init.d/gitlab "GitLab init service file" [gl-recipes]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/init "GitLab Recipes repository" diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 8f7280d5128..604dff5983d 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -10,43 +10,43 @@ an SMTP server, but you're not seeing mail delivered. Here's how to check the se 1. Run a Rails console: - ```sh - sudo gitlab-rails console production - ``` + ```sh + sudo gitlab-rails console production + ``` - or for source installs: + or for source installs: - ```sh - bundle exec rails console production - ``` + ```sh + bundle exec rails console production + ``` 1. Look at the ActionMailer `delivery_method` to make sure it matches what you intended. If you configured SMTP, it should say `:smtp`. If you're using Sendmail, it should say `:sendmail`: - ```ruby - irb(main):001:0> ActionMailer::Base.delivery_method - => :smtp - ``` + ```ruby + irb(main):001:0> ActionMailer::Base.delivery_method + => :smtp + ``` 1. If you're using SMTP, check the mail settings: - ```ruby - irb(main):002:0> ActionMailer::Base.smtp_settings - => {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true}``` - ``` + ```ruby + irb(main):002:0> ActionMailer::Base.smtp_settings + => {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true}``` + ``` - In the example above, the SMTP server is configured for the local machine. If this is intended, you may need to check your local mail - logs (e.g. `/var/log/mail.log`) for more details. + In the example above, the SMTP server is configured for the local machine. If this is intended, you may need to check your local mail + logs (e.g. `/var/log/mail.log`) for more details. -1. Send a test message via the console. +1. Send a test message via the console. - ```ruby - irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now - ``` + ```ruby + irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now + ``` - If you do not receive an e-mail and/or see an error message, then check - your mail server settings. + If you do not receive an e-mail and/or see an error message, then check + your mail server settings. ## Advanced Issues @@ -103,37 +103,37 @@ downtime. Otherwise skip to the next section. 1. Run `sudo gdb -p <PID>` to attach to the unicorn process. 1. In the gdb window, type: - ``` - call (void) rb_backtrace() - ``` + ``` + call (void) rb_backtrace() + ``` 1. This forces the process to generate a Ruby backtrace. Check `/var/log/gitlab/unicorn/unicorn_stderr.log` for the backtace. For example, you may see: - ```ruby - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' - ``` + ```ruby + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' + ``` 1. To see the current threads, run: - ``` - thread apply all bt - ``` + ``` + thread apply all bt + ``` 1. Once you're done debugging with `gdb`, be sure to detach from the process and exit: - ``` - detach - exit - ``` + ``` + detach + exit + ``` Note that if the unicorn process terminates before you are able to run these commands, gdb will report an error. To buy more time, you can always raise the @@ -162,21 +162,21 @@ separate Rails process to debug the issue: 1. Create a Personal Access Token for your user (Profile Settings -> Access Tokens). 1. Bring up the GitLab Rails console. For omnibus users, run: - ``` - sudo gitlab-rails console - ``` + ``` + sudo gitlab-rails console + ``` 1. At the Rails console, run: - ```ruby - [1] pry(main)> app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' - ``` + ```ruby + [1] pry(main)> app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' + ``` - For example: + For example: - ```ruby - [1] pry(main)> app.get 'https://gitlab.com/gitlab-org/gitlab-ce/issues/1?private_token=123456' - ``` + ```ruby + [1] pry(main)> app.get 'https://gitlab.com/gitlab-org/gitlab-ce/issues/1?private_token=123456' + ``` 1. In a new window, run `top`. It should show this ruby process using 100% CPU. Write down the PID. 1. Follow step 2 from the previous section on using gdb. @@ -209,7 +209,7 @@ ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -s 10 The output in `/tmp/unicorn.txt` may help diagnose the root cause. -# More information +## More information - [Debugging Stuck Ruby Processes](https://blog.newrelic.com/2013/04/29/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/) - [Cheatsheet of using gdb and ruby processes](gdb-stuck-ruby.txt) diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md new file mode 100644 index 00000000000..ab3b25f0e97 --- /dev/null +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -0,0 +1,27 @@ +--- +type: reference +--- + +# Diagnostics tools + +These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. +They are listed here for transparency, and they may be useful for users with experience +with troubleshooting GitLab. If you are currently having an issue with GitLab, you +may want to check your [support options](https://about.gitlab.com/support/) first, +before attempting to use these tools. + +## gitlabsos + +The [gitlabsos](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) utility +provides a unified method of gathering info and logs from GitLab and the system it's +running on. + +## strace-parser + +[strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze +and summarize raw strace data. + +## Pritaly + +[Pritaly](https://gitlab.com/wchandler/pritaly) takes Gitaly logs and colorizes output +or converts the logs to JSON. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md new file mode 100644 index 00000000000..c4a7ba01fae --- /dev/null +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -0,0 +1,345 @@ +# Troubleshooting ElasticSearch + +Troubleshooting ElasticSearch requires: + +- Knowledge of common terms. +- Establishing within which category the problem fits. + +## Common terminology + +- **Lucene**: A full-text search library written in Java. +- **Near Realtime (NRT)**: Refers to the slight latency from the time to index a + document to the time when it becomes searchable. +- **Cluster**: A collection of one or more nodes that work together to hold all + the data, providing indexing and search capabilities. +- **Node**: A single server that works as part of a cluster. +- **Index**: A collection of documents that have somewhat similar characteristics. +- **Document**: A basic unit of information that can be indexed. +- **Shards**: Fully-functional and independent subdivisions of indices. Each shard is actually + a Lucene index. +- **Replicas**: Failover mechanisms that duplicate indices. + +## Troubleshooting workflows + +The type of problem will determine what steps to take. The possible troubleshooting workflows are for: + +- Search results. +- Indexing. +- Integration. +- Performance. + +### Search Results workflow + +The following workflow is for ElasticSearch search results issues: + +```mermaid +graph TD; + B --> |No| B1 + B --> |Yes| B4 + B1 --> B2 + B2 --> B3 + B4 --> B5 + B5 --> |Yes| B6 + B5 --> |No| B7 + B7 --> B8 + B{Is GitLab using<br>ElasticSearch for<br>searching?} + B1[Check Admin Area > Integrations<br>to ensure the settings are correct] + B2[Perform a search via<br>the rails console] + B3[If all settings are correct<br>and it still doesn't show ElasticSearch<br>doing the searches, escalate<br>to GitLab support.] + B4[Perform<br>the same search via the<br>ElasticSearch API] + B5{Are the results<br>the same?} + B6[This means it is working as intended.<br>Speak with GitLab support<br>to confirm if the issue lies with<br>the filters.] + B7[Check the index status of the project<br>containing the missing search<br>results.] + B8(Indexing Troubleshooting) +``` + +### Indexing workflow + +The following workflow is for ElasticSearch indexing issues: + +```mermaid +graph TD; + C --> |Yes| C1 + C1 --> |Yes| C2 + C1 --> |No| C3 + C3 --> |Yes| C4 + C3 --> |No| C5 + C --> |No| C6 + C6 --> |No| C10 + C7 --> |GitLab| C8 + C7 --> |ElasticSearch| C9 + C6 --> |Yes| C7 + C10 --> |No| C12 + C10 --> |Yes| C11 + C12 --> |Yes| C13 + C12 --> |No| C14 + C14 --> |Yes| C15 + C14 --> |No| C16 + C{Is the problem with<br>creating an empty<br>index?} + C1{Does the gitlab-production<br>index exist on the<br>ElasticSearch instance?} + C2(Try to manually<br>delete the index on the<br>ElasticSearch instance and<br>retry creating an empty index.) + C3{Can indices be made<br>manually on the ElasticSearch<br>instance?} + C4(Retry the creation of an empty index) + C5(It is best to speak with an<br>ElasticSearch admin concerning the<br>instance's inability to create indices.) + C6{Is the indexer presenting<br>errors during indexing?} + C7{Is the error a GitLab<br>error or an ElasticSearch<br>error?} + C8[Escalate to<br>GitLab support] + C9[You will want<br>to speak with an<br>ElasticSearch admin.] + C10{Does the index status<br>show 100%?} + C11[Escalate to<br>GitLab support] + C12{Does re-indexing the project<br> present any GitLab errors?} + C13[Rectify the GitLab errors and<br>restart troubleshooting, or<br>escalate to GitLab support.] + C14{Does re-indexing the project<br>present errors on the <br>ElasticSearch instance?} + C15[It would be best<br>to speak with an<br>ElasticSearch admin.] + C16[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] +``` + +### Integration workflow + +The following workflow is for ElasticSearch integration issues: + +```mermaid +graph TD; + D --> |No| D1 + D --> |Yes| D2 + D2 --> |No| D3 + D2 --> |Yes| D4 + D4 --> |No| D5 + D4 --> |Yes| D6 + D{Is the error concerning<br>the beta indexer?} + D1[It would be best<br>to speak with an<br>ElasticSearch admin.] + D2{Is the ICU development<br>package installed?} + D3>This package is required.<br>Install the package<br>and retry.] + D4{Is the error stemming<br>from the indexer?} + D5[This would indicate an OS level<br> issue. It would be best to<br>contact your sysadmin.] + D6[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] +``` + +### Performance workflow + +The following workflow is for ElasticSearch performance issues: + +```mermaid +graph TD; + F --> |Yes| F1 + F --> |No| F2 + F2 --> |No| F3 + F2 --> |Yes| F4 + F4 --> F5 + F5 --> |No| F6 + F5 --> |Yes| F7 + F{Is the ElasticSearch instance<br>running on the same server<br>as the GitLab instance?} + F1(This is not advised and will cause issues.<br>We recommend moving the ElasticSearch<br>instance to a different server.) + F2{Does the ElasticSearch<br>server have at least 8<br>GB of RAM and 2 CPU<br>cores?} + F3(According to ElasticSearch, a non-prod<br>server needs these as a base requirement.<br>Production often requires more. We recommend<br>you increase the server specifications.) + F4(Obtain the <br>cluster health information) + F5(Does it show the<br>status as green?) + F6(We recommend you speak with<br>an ElasticSearch admin<br>about implementing sharding.) + F7(Escalate to<br>GitLab support.) +``` + +## Troubleshooting walkthrough + +Most ElasticSearch troubleshooting can be broken down into 4 categories: + +- [Troubleshooting search results](#troubleshooting-search-results) +- [Troubleshooting indexing](#troubleshooting-indexing) +- [Troubleshooting integration](#troubleshooting-integration) +- [Troubleshooting performance](#troubleshooting-performance) + +Generally speaking, if it does not fall into those four categories, it is either: + +- Something GitLab support needs to look into. +- Not a true ElasticSearch issue. + +Exercise caution. Issues that appear to be ElasticSearch problems can be OS-level issues. + +### Troubleshooting search results + +Troubleshooting search result issues is rather straight forward on ElasticSearch. + +The first step is to confirm GitLab is using ElasticSearch for the search function. +To do this: + +1. Confirm the integration is enabled in **Admin Area > Settings > Integrations**. +1. Confirm searches utilize ElasticSearch by accessing the rails console + (`sudo gitlab-rails console`) and running the following commands: + + ```rails + u = User.find_by_email('email_of_user_doing_search') + s = SearchService.new(u, {:search => 'search_term'}) + pp s.search_objects.class.name + ``` + +The ouput from the last command is the key here. If it shows: + +- `ActiveRecord::Relation`, **it is not** using ElasticSearch. +- `Kaminari::PaginatableArray`, **it is** using ElasticSearch. + +| Not using ElasticSearch | Using ElasticSearch | +|--------------------------|------------------------------| +| `ActiveRecord::Relation` | `Kaminari::PaginatableArray` | + +If all the settings look correct and it is still not using ElasticSearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue. + +Moving past that, it is best to attempt the same search using the [ElasticSearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and compare the results from what you see in GitLab. + +If the results: + +- Sync up, then there is not a technical "issue" per se. Instead, it might be a problem + with the ElasticSearch filters we are using. This can be complicated, so it is best to + escalate to GitLab support to check these and guide you on the potential on whether or + not a feature request is needed. +- Do not match up, this indicates a problem with the documents generated from the + project. It is best to re-index that project and proceed with + [Troubleshooting indexing](#troubleshooting-indexing). + +### Troubleshooting indexing + +Troubleshooting indexing issues can be tricky. It can pretty quickly go to either GitLab +support or your ElasticSearch admin. + +The best place to start is to determine if the issue is with creating an empty index. +If it is, check on the ElasticSearch side to determine if the `gitlab-production` (the +name for the GitLab index) exists. If it exists, manually delete it on the ElasticSearch +side and attempt to recreate it from the +[`create_empty_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) +rake task. + +If you still encounter issues, try creating an index manually on the ElasticSearch +instance. The details of the index aren't important here, as we want to test if indices +can be made. If the indices: + +- Cannot be made, speak with your ElasticSearch admin. +- Can be made, Escalate this to GitLab support. + +If the issue is not with creating an empty index, the next step is to check for errors +during the indexing of projects. If errors do occur, they will either stem from the indexing: + +- On the GitLab side. You need to rectify those. If they are not + something you are familiar with, contact GitLab support for guidance. +- Within the ElasticSearch instance itself. See if the error is [documented and has a fix](../../integration/elasticsearch.md#troubleshooting). If not, speak with your ElasticSearch admin. + +If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following rake tasks: + +- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows the overall status) +- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows specific projects that are not indexed) + +If: + +- Everything is showing at 100%, escalate to GitLab support. This could be a potential + bug/issue. +- You do see something not at 100%, attempt to reindex that project. To do this, + run `sudo gitlab-rake gitlab:elastic:index_projects ID_FROM=<project ID> ID_TO=<project ID>`. + +If reindexing the project shows: + +- Errors on the GitLab side, escalate those to GitLab support. +- ElasticSearch errors or doesn't present any errors at all, reach out to your + ElasticSearch admin to check the instance. + +### Troubleshooting integration + +Troubleshooting integration tends to be pretty straight forward, as there really isn't +much to "integrate" here. + +If the issue is: + +- Not concerning the beta indexer, it is almost always an + ElasticSearch-side issue. This means you should reach out to your ElasticSearch admin + regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach + out to GitLab support. +- With the beta indexer, check if the ICU development package is installed. + This is a required package so make sure you install it. + +Beyond that, you will want to review the error. If it is: + +- Specifically from the indexer, this could be a bug/issue and should be escalated to + GitLab support. +- An OS issue, you will want to reach out to your systems administrator. + +### Troubleshooting performance + +Troubleshooting performance can be difficult on ElasticSearch. There is a ton of tuning +that *can* be done, but the majority of this falls on shoulders of a skilled +ElasticSearch administrator. + +Generally speaking, ensure: + +* The ElasticSearch server **is not** running on the same node as GitLab. +* The ElasticSearch server have enough RAM and CPU cores. +* That sharding **is** being used. + +Going into some more detail here, if ElasticSearch is running on the same server as GitLab, resource contention is **very** likely to occur. Ideally, ElasticSearch, which requires ample resources, should be running on its own server (maybe coupled with logstash and kibana). + +When it comes to ElasticSearch, RAM is the key resource. ElasticSearch themselves recommend: + +- **At least** 8 GB of RAM for a non-production instance. +- **At least** 16 GB of RAM for a production instance. +- Ideally, 64 GB of RAM. + +For CPU, ElasticSearch recommends at least 2 CPU cores, but ElasticSearch states common +setups use up to 8 cores. For more details on server specs, check out +[ElasticSearch's hardware guide](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html). + +Beyond the obvious, sharding comes into play. Sharding is a core part of ElasticSearch. +It allows for horizontal scaling of indices, which is helpful when you are dealing with +a large amount of data. + +With the way GitLab does indexing, there is a **huge** amount of documents being +indexed. By utilizing sharding, you can speed up ElasticSearch's ability to locate +data, since each shard is a Lucene index. + +If you are not using sharding, you are likely to hit issues when you start using +ElasticSearch in a production environment. + +Keep in mind that an index with only one shard has **no scale factor** and will +likely encounter issues when called upon with some frequency. + +If you need to know how many shards, read +[ElasticSearch's documentation on capacity planning](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/capacity-planning.html), +as the answer is not straight forward. + +The easiest way to determine if sharding is in use is to check the output of the +[ElasticSearch Health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html): + +- Red means the cluster is down. +- Yellow means it is up with no sharding/replication. +- Green means it is healthy (up, sharding, replicating). + +For production use, it should always be green. + +Beyond these steps, you get into some of the more complicated things to check, +such as merges and caching. These can get complicated and it takes some time to +learn them, so it is best to escalate/pair with an ElasticSearch expert if you need to +dig further into these. + +Feel free to reach out to GitLab support, but this is likely to be something a skilled +ElasticSearch admin has more experience with. + +## Common issues + +All common issues [should be documented](../../integration/elasticsearch.md#troubleshooting). If not, +feel free to update that page with issues you encounter and solutions. + +## Replication + +Setting up ElasticSearch isn't too bad, but it can be a bit finnicky and time consuming. + +The eastiest method is to spin up a docker container with the required version and +bind ports 9200/9300 so it can be used. + +The following is an example of running a docker container of ElasticSearch v7.2.0: + +```bash +docker pull docker.elastic.co/elasticsearch/elasticsearch:7.2.0 +docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.2.0 +``` + +From here, you can: + +- Grab the IP of the docker container (use `docker inspect <container_id>`) +- Use `<IP.add.re.ss:9200>` to communicate with it. + +This is a quick method to test out ElasticSearch, but by no means is this a +production solution. diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md new file mode 100644 index 00000000000..95cdb1508fa --- /dev/null +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -0,0 +1,251 @@ +--- +type: reference +--- + +# Kubernetes, GitLab and You + +This is a list of useful information regarding Kubernetes that the GitLab Support +Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone +can make use of the Support team's collected knowledge + +CAUTION: **Caution:** +These commands **can alter or break** your Kubernetes components so use these at your own risk. + +If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how +to use these commands, it is best to [contact Support](https://about.gitlab.com/support/) +and they will assist you with any issues you are having. + +## Generic kubernetes commands + +- How to authorize to your GCP project (can be especially useful if you have projects + under different GCP accounts): + + ```bash + gcloud auth login + ``` + +- How to access Kubernetes dashboard: + + ```bash + # for minikube: + minikube dashboard —url + # for non-local installations if access via kubectl is configured: + kubectl proxy + ``` + +- How to ssh to a Kubernetes node and enter the container as root + <https://github.com/kubernetes/kubernetes/issues/30656>: + + - For GCP, you may find the node name and run `gcloud compute ssh node-name`. + - List containers using `docker ps`. + - Enter container using `docker exec --user root -ti container-id bash`. + +- How to copy a file from local machine to a pod: + + ```bash + kubectl cp file-name pod-name:./destination-path + ``` + +- What to do with pods in `CrashLoopBackoff` status: + + - Check logs via Kubernetes dashboard. + - Check logs via `kubectl`: + + ```bash + kubectl logs <unicorn pod> -c dependencies + ``` + +- How to tail all Kubernetes cluster events in real time: + + ```bash + kubectl get events -w --all-namespaces + ``` + +- How to get logs of the previously terminated pod instance: + + ```bash + kubectl logs <pod-name> --previous + ``` + + NOTE: **Note:** + No logs are kept in the containers/pods themselves, everything is written to stdout. + This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) + for details. + +## Gitlab-specific kubernetes information + +- Minimal config that can be used to test a Kubernetes helm chart can be found + [here](https://gitlab.com/charts/gitlab/issues/620). + +- Tailing logs of a separate pod. An example for a unicorn pod: + + ```bash + kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn + ``` + +- It is not possible to get all the logs via `kubectl` at once, like with `gitlab-ctl tail`, + but a number of third-party tools can be used to do it: + + - [Kubetail](https://github.com/johanhaleby/kubetail) + - [kail: kubernetes tail](https://github.com/boz/kail) + - [stern](https://github.com/wercker/stern) + +- Check all events in the `gitlab` namespace (the namespace name can be different if you + specified a different one when deploying the helm chart): + + ```bash + kubectl get events -w --namespace=gitlab + ``` + +- Most of the useful GitLab tools (console, rake tasks, etc) are found in the task-runner + pod. You may enter it and run commands inside or run them from the outside: + + ```bash + # find the pod + kubectl get pods | grep task-runner + + # enter it + kubectl exec -it <task-runner-pod-name> bash + + # open rails console + # rails console can be also called from other GitLab pods + /srv/gitlab/bin/rails console + + # source-style commands should also work + /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production + + # run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart + /usr/local/bin/gitlab-rake gitlab:check + + # open console without entering pod + kubectl exec -it <task-runner-pod-name> /srv/gitlab/bin/rails console + + # check the status of DB migrations + kubectl exec -it <task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status + ``` + + You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. + +- Troubleshooting **Operations > Kubernetes** integration: + + - Check the output of `kubectl get events -w --all-namespaces`. + - Check the logs of pods within `gitlab-managed-apps` namespace. + - On the side of GitLab check sidekiq log and kubernetes log. When GitLab is installed + via helm chart, kubernetes.log can be found inside the sidekiq pod. + +- How to get your initial admin password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: + + ```bash + # find the name of the secret containing the password + kubectl get secrets | grep initial-root + # decode it + kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo + ``` + +- How to connect to a GitLab postgres database: + + ```bash + kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole -p + ``` + +- How to get info about helm installation status: + + ```bash + helm status name-of-installation + ``` + +- How to update GitLab installed using helm chart: + + ```bash + helm repo upgrade + + # get current values and redirect them to yaml file (analogue of gitlab.rb values) + helm get values <release name> > gitlab.yaml + + # run upgrade itself + helm upgrade <release name> <chart path> -f gitlab.yaml + ``` + + After <https://canary.gitlab.com/charts/gitlab/issues/780> is fixed, it should + be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#updating-gitlab-using-the-helm-chart) + for upgrades. + +- How to apply changes to GitLab config: + + - Modify the `gitlab.yaml` file. + - Run the following command to apply changes: + + ```bash + helm upgrade <release name> <chart path> -f gitlab.yaml + ``` + +## Installation of minimal GitLab config via minukube on macOS + +This section is based on [Developing for Kubernetes with Minikube](https://gitlab.com/charts/gitlab/blob/master/doc/minikube/index.md) +and [Helm](https://gitlab.com/charts/gitlab/blob/master/doc/helm/index.md). Refer +to those documents for details. + +- Install kubectl via Homebrew: + + ```bash + brew install kubernetes-cli + ``` + +- Install minikube via Homebrew: + + ```bash + brew cask install minikube + ``` + +- Start minikube and configure it. If minikube cannot start, try running `minikube delete && minikube start` + and repeat the steps: + + ```bash + minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work + minikube addons enable ingress + minikube addons enable kube-dns + ``` + +- Install helm via Homebrew and initialize it: + + ```bash + brew install kubernetes-helm + helm init --service-account tiller + ``` + +- Copy the file <https://gitlab.com/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml> + to your workstation. + +- Find the IP address in the output of `minikube ip` and update the yaml file with + this IP address. + +- Install the GitLab helm chart: + + ```bash + helm repo add gitlab https://charts.gitlab.io + helm install --name gitlab -f <path-to-yaml-file> gitlab/gitlab + ``` + + If you want to modify some GitLab settings, you can use the above-mentioned config + as a base and create your own yaml file. + +- Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. + The installation could take up to 20-30 minutes depending on the amount of resources + on your workstation. + +- When all the pods show either a `Running` or `Completed` status, get the GitLab password as + described in [Initial login](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#initial-login), + and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain` + where `domain` is the value provided in the yaml file. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 708b59a273b..99f1c386183 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -1,21 +1,18 @@ # Uploads administration ->**Notes:** Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled. -### Using local storage +## Using local storage ->**Notes:** +NOTE: **Note:** This is the default configuration To change the location where the uploads are stored locally, follow the steps below. ---- - **In Omnibus installations:** ->**Notes:** +NOTE: **Note:** For historical reasons, uploads are stored into a base directory, which by default is `uploads/-/system`. It is strongly discouraged to change this configuration option on an existing GitLab installation. _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ @@ -23,15 +20,13 @@ _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ 1. To change the storage path for example to `/mnt/storage/uploads`, edit `/etc/gitlab/gitlab.rb` and add the following line: - ```ruby - gitlab_rails['uploads_storage_path'] = "/mnt/storage/" - gitlab_rails['uploads_base_dir'] = "uploads" - ``` + ```ruby + gitlab_rails['uploads_storage_path'] = "/mnt/storage/" + gitlab_rails['uploads_base_dir'] = "uploads" + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **In installations from source:** _The uploads are stored by default in @@ -40,15 +35,15 @@ _The uploads are stored by default in 1. To change the storage path for example to `/mnt/storage/uploads`, edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - uploads: - storage_path: /mnt/storage - base_dir: uploads - ``` + ```yaml + uploads: + storage_path: /mnt/storage + base_dir: uploads + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. -### Using object storage **[CORE ONLY]** +## Using object storage **(CORE ONLY)** > **Notes:** > @@ -60,7 +55,7 @@ If you don't want to use the local disk where GitLab is installed to store the uploads, you can use an object storage provider like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. -### Object Storage Settings +## Object Storage Settings For source installations the following settings are nested under `uploads:` and then `object_store:`. On omnibus installs they are prefixed by `uploads_object_store_`. @@ -73,7 +68,7 @@ For source installations the following settings are nested under `uploads:` and | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | -#### S3 compatible connection settings +### S3 compatible connection settings The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: @@ -83,6 +78,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -97,27 +93,104 @@ _The uploads are stored by default in 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: - ```ruby - gitlab_rails['uploads_object_store_enabled'] = true - gitlab_rails['uploads_object_store_remote_directory'] = "uploads" - gitlab_rails['uploads_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', - 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' - } - ``` - - >**Note:** - If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. - - ```ruby - gitlab_rails['uploads_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'use_iam_profile' => true - } - ``` + ```ruby + gitlab_rails['uploads_object_store_enabled'] = true + gitlab_rails['uploads_object_store_remote_directory'] = "uploads" + gitlab_rails['uploads_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: **Note:** + If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. + + ```ruby + gitlab_rails['uploads_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). + +**In installations from source:** + +_The uploads are stored by default in +`/home/git/gitlab/public/uploads/-/system`._ + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + uploads: + object_store: + enabled: true + remote_directory: "uploads" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). + +### Oracle Cloud S3 connection settings + +Note that Oracle Cloud S3 must be sure to use the following settings: + +| Setting | Value | +|---------|-------| +| `enable_signature_v4_streaming` | false | +| `path_style` | true | + +If `enable_signature_v4_streaming` is set to `true`, you may see the +following error: + +``` +STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported +``` + +### OpenStack compatible connection settings + +The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: + +| Setting | Description | Default | +|---------|-------------|---------| +| `provider` | Always `OpenStack` for compatible hosts | OpenStack | +| `openstack_username` | OpenStack username | | +| `openstack_api_key` | OpenStack api key | | +| `openstack_temp_url_key` | OpenStack key for generating temporary urls | | +| `openstack_auth_url` | OpenStack authentication endpont | | +| `openstack_region` | OpenStack region | | +| `openstack_tenant` | OpenStack tenant ID | + +**In Omnibus installations:** + +_The uploads are stored by default in +`/var/opt/gitlab/gitlab-rails/public/uploads/-/system`._ + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['uploads_object_store_remote_directory'] = "OPENSTACK_OBJECT_CONTAINER_NAME" + gitlab_rails['uploads_object_store_connection'] = { + 'provider' => 'OpenStack', + 'openstack_username' => 'OPENSTACK_USERNAME', + 'openstack_api_key' => 'OPENSTACK_PASSWORD', + 'openstack_temp_url_key' => 'OPENSTACK_TEMP_URL_KEY', + 'openstack_auth_url' => 'https://auth.cloud.ovh.net/v2.0/', + 'openstack_region' => 'DE1', + 'openstack_tenant' => 'TENANT_ID', + } + ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). @@ -132,19 +205,25 @@ _The uploads are stored by default in 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - uploads: - object_store: - enabled: true - remote_directory: "uploads" # The bucket name - connection: - provider: AWS # Only AWS supported at the moment - aws_access_key_id: AWS_ACESS_KEY_ID - aws_secret_access_key: AWS_SECRET_ACCESS_KEY - region: eu-central-1 - ``` + ```yaml + uploads: + object_store: + enabled: true + direct_upload: false + background_upload: true + proxy_download: false + remote_directory: OPENSTACK_OBJECT_CONTAINER_NAME + connection: + provider: OpenStack + openstack_username: OPENSTACK_USERNAME + openstack_api_key: OPENSTACK_PASSWORD + openstack_temp_url_key: OPENSTACK_TEMP_URL_KEY + openstack_auth_url: 'https://auth.cloud.ovh.net/v2.0/' + openstack_region: DE1 + openstack_tenant: 'TENANT_ID' + ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab][] for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" |
