diff options
| author | Achilleas Pipinellis <axil@gitlab.com> | 2019-08-20 20:22:43 +0200 |
|---|---|---|
| committer | Achilleas Pipinellis <axil@gitlab.com> | 2019-08-20 20:22:43 +0200 |
| commit | e61308ce1d82e12e5087371469baea4a452875d1 (patch) | |
| tree | 62551a3ae4eab75e5af7e3b35358c07a51b2132f /doc/ci | |
| parent | 4f323bb62fbe71a4352de25cab141f361a3fe1a6 (diff) | |
| parent | 2989ed078c1d45b0959dcecb1bc3c8f4740a3c0d (diff) | |
| download | gitlab-ce-docs-patch-71.tar.gz | |
Merge branch 'master' into docs-patch-71docs-patch-71
Diffstat (limited to 'doc/ci')
75 files changed, 1961 insertions, 845 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 1743c38eb46..ca9d0aa61bd 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -13,6 +13,11 @@ through the [continuous methodologies](introduction/index.md#introduction-to-cic - Continuous Delivery (CD) - Continuous Deployment (CD) +NOTE: **Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more.** +Watch our +["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) +webcast to learn about continuous methods and how GitLab’s built-in CI can help you simplify and scale software development. + ## Overview Continuous Integration works by pushing small code chunks to your @@ -48,6 +53,9 @@ the following documents: - [GitLab CI/CD basic workflow](introduction/index.md#basic-cicd-workflow). - [Step-by-step guide for writing `.gitlab-ci.yml` for the first time](../user/project/pages/getting_started_part_four.md). +If you're coming over from Jenkins, you can also check out our handy [reference](jenkins/index.md) +for converting your pipelines. + You can also get started by using one of the [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/ci/templates) available through the UI. You can use them by creating a new file, @@ -59,11 +67,11 @@ to your needs: For a broader overview, see the [CI/CD getting started](quick_start/README.md) guide. Once you're familiar with how GitLab CI/CD works, see the -[`. gitlab-ci.yml` full reference](yaml/README.md) +[`.gitlab-ci.yml` full reference](yaml/README.md) for all the attributes you can set and use. NOTE: **Note:** -GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/admin_area/settings/continuous_integration.md#extra-shared-runners-pipeline-minutes-quota). +GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/gitlab_com/index.md#shared-runners). ## Configuration @@ -81,6 +89,7 @@ GitLab CI/CD supports numerous configuration options: | [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules.| | [SSH keys for CI/CD](ssh_keys/README.md) | Using SSH keys in your CI pipelines. | | [Pipelines triggers](triggers/README.md) | Trigger pipelines through the API. | +| [Pipelines for Merge Requests](merge_request_pipelines/index.md) | Design a pipeline structure for running a pipeline in merge requests. | | [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | | [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own GitLab Runners to execute your scripts. | | [Optimize GitLab and Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repos. | @@ -103,27 +112,27 @@ Its feature set is listed on the table below according to DevOps stages. | **Verify** || | [Browser Performance Testing](../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the performance impact of pending code changes. | | [CI services](services/README.md) | Link Docker containers with your base image.| -| [Code Quality](../user/project/merge_requests/code_quality.md) **[STARTER]** | Analyze your source code quality. | -| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | -| [Interactive Web Terminals](interactive_web_terminal/index.md) **[CORE ONLY]** | Open an interactive web terminal to debug the running jobs. | +| [Code Quality](../user/project/merge_requests/code_quality.md) **(STARTER)** | Analyze your source code quality. | +| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **(PREMIUM)** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | +| [Interactive Web Terminals](interactive_web_terminal/index.md) **(CORE ONLY)** | Open an interactive web terminal to debug the running jobs. | | [JUnit tests](junit_test_reports.md) | Identify script failures directly on merge requests. | | [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | |---+---| | **Release** || | [Auto Deploy](../topics/autodevops/index.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | | [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | -| [Canary Deployments](../user/project/canary_deployments.md) **[PREMIUM]** | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | -| [Deploy Boards](../user/project/deploy_boards.md) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | -| [Feature Flags](../user/project/operations/feature_flags.md) **[PREMIUM]** | Deploy your features behind Feature Flags. | +| [Canary Deployments](../user/project/canary_deployments.md) **(PREMIUM)** | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | +| [Deploy Boards](../user/project/deploy_boards.md) **(PREMIUM)** | Check the current health and status of each CI/CD environment running on Kubernetes. | +| [Feature Flags](../user/project/operations/feature_flags.md) **(PREMIUM)** | Deploy your features behind Feature Flags. | | [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | | [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | | [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | |---+---| | **Secure** || -| [Container Scanning](../user/application_security/container_scanning/index.md) **[ULTIMATE]** | Check your Docker containers for known vulnerabilities.| -| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | -| [License Management](../user/application_security/license_management/index.md) **[ULTIMATE]** | Search your project dependencies for their licenses. | -| [Security Test reports](../user/project/merge_requests/index.md#security-reports-ultimate) **[ULTIMATE]** | Check for app vulnerabilities. | +| [Container Scanning](../user/application_security/container_scanning/index.md) **(ULTIMATE)** | Check your Docker containers for known vulnerabilities.| +| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | +| [License Management](../user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | +| [Security Test reports](../user/project/merge_requests/index.md#security-reports-ultimate) **(ULTIMATE)** | Check for app vulnerabilities. | ## Examples @@ -132,7 +141,7 @@ on the [CI Examples](examples/README.md) page. GitLab also provides [example projects](https://gitlab.com/gitlab-examples) pre-configured to use GitLab CI/CD. -## Administration **[CORE ONLY]** +## Administration **(CORE ONLY)** As a GitLab administrator, you can change the default behavior of GitLab CI/CD for: @@ -154,6 +163,7 @@ for your CI/CD infrastructure: - [Why we chose GitLab CI for our CI/CD solution](https://about.gitlab.com/2016/10/17/gitlab-ci-oohlala/) - [Building our web-app on GitLab CI](https://about.gitlab.com/2016/07/22/building-our-web-app-on-gitlab-ci/) +- [5 Teams that made the switch to GitLab CI/CD](https://about.gitlab.com/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/) See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 9a5a3624c73..f8151e3e18c 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -8,7 +8,7 @@ GitLab CI/CD provides a caching mechanism that can be used to save time when your jobs are running. Caching is about speeding the time a job is executed by reusing the same -content of a previous job. It can be particularly useful when your are +content of a previous job. It can be particularly useful when you are developing software that depends on other libraries which are fetched via the internet during build time. @@ -378,8 +378,8 @@ Here's what happens behind the scenes: 1. `script` is executed. 1. `after_script` is executed. 1. `cache` runs and the `vendor/` directory is zipped into `cache.zip`. - This file is then saved in the directory based on the - [Runner's setting](#where-the-caches-are-stored) and the `cache: key`. + This file is then saved in the directory based on the + [Runner's setting](#where-the-caches-are-stored) and the `cache: key`. 1. `job B` runs. 1. The cache is extracted (if found). 1. `before_script` is executed. @@ -520,7 +520,7 @@ via GitLab's UI: 1. Navigate to your project's **CI/CD > Pipelines** page. 1. Click on the **Clear Runner caches** button to clean up the cache. -  +  1. On the next push, your CI/CD job will use a new cache. diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md index 241134783da..29d4f93f02e 100644 --- a/doc/ci/chatops/README.md +++ b/doc/ci/chatops/README.md @@ -4,14 +4,14 @@ type: index, concepts, howto # GitLab ChatOps -> **Notes:** -> -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4466) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24780) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.9. -> -> - ChatOps is currently in alpha, with some important features missing like access control. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4466) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24780) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.9. GitLab ChatOps provides a method to interact with CI/CD jobs through chat services like Slack. Many organizations' discussion, collaboration, and troubleshooting is taking place in chat services these days, and having a method to run CI/CD jobs with output posted back to the channel can significantly augment a team's workflow. +NOTE: **Note:** +ChatOps is currently in alpha with some important features missing, like access control. + ## How it works GitLab ChatOps is built upon two existing features: diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index bbb25c78ec5..54b21939116 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -2,7 +2,7 @@ type: howto --- -# Using GitLab CI/CD with a Bitbucket Cloud repository **[PREMIUM]** +# Using GitLab CI/CD with a Bitbucket Cloud repository **(PREMIUM)** GitLab CI/CD can be used with Bitbucket Cloud by: @@ -14,9 +14,9 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: 1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and create the project. -  +  - GitLab will import the repository and enable [Pull Mirroring][pull-mirroring]. + GitLab will import the repository and enable [Pull Mirroring][pull-mirroring]. 1. In GitLab create a [Personal Access Token](../../user/profile/personal_access_tokens.md) @@ -26,127 +26,127 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: 1. In Bitbucket, from **Settings > Webhooks**, create a new web hook to notify GitLab of new commits. - The web hook URL should be set to the GitLab API to trigger pull mirroring, - using the Personal Access Token we just generated for authentication. + The web hook URL should be set to the GitLab API to trigger pull mirroring, + using the Personal Access Token we just generated for authentication. - ```text - https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> - ``` + ```text + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` - The web hook Trigger should be set to 'Repository Push'. + The web hook Trigger should be set to 'Repository Push'. -  +  - After saving, test the web hook by pushing a change to your Bitbucket - repository. + After saving, test the web hook by pushing a change to your Bitbucket + repository. 1. In Bitbucket, create an **App Password** from **Bitbucket Settings > App Passwords** to authenticate the build status script setting commit build statuses in Bitbucket. Repository write permissions are required. -  +  1. In GitLab, from **Settings > CI/CD > Environment variables**, add variables to allow communication with Bitbucket via the Bitbucket API: - `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above. + `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above. - `BITBUCKET_USERNAME`: the username of the Bitbucket account. + `BITBUCKET_USERNAME`: the username of the Bitbucket account. - `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ. + `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ. - `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ. + `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ. 1. In Bitbucket, add a script to push the pipeline status to Bitbucket. - > Note: changes made in GitLab will be overwritten by any changes made - > upstream in Bitbucket. - - Create a file `build_status` and insert the script below and run - `chmod +x build_status` in your terminal to make the script executable. - - ```bash - #!/usr/bin/env bash - - # Push GitLab CI/CD build status to Bitbucket Cloud - - if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then - echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set" - exit 1 - fi - if [ -z "$BITBUCKET_USERNAME" ]; then - echo "ERROR: BITBUCKET_USERNAME is not set" - exit 1 - fi - if [ -z "$BITBUCKET_NAMESPACE" ]; then - echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE" - BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE - fi - if [ -z "$BITBUCKET_REPOSITORY" ]; then - echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME" - BITBUCKET_REPOSITORY=$CI_PROJECT_NAME - fi - - BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0" - BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build" - BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME" - - case "$BUILD_STATUS" in - running) - BITBUCKET_STATE="INPROGRESS" - BITBUCKET_DESCRIPTION="The build is running!" - ;; - passed) - BITBUCKET_STATE="SUCCESSFUL" - BITBUCKET_DESCRIPTION="The build passed!" - ;; - failed) - BITBUCKET_STATE="FAILED" - BITBUCKET_DESCRIPTION="The build failed." - ;; - esac - - echo "Pushing status to $BITBUCKET_STATUS_API..." - curl --request POST $BITBUCKET_STATUS_API \ - --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \ - --header "Content-Type:application/json" \ - --silent \ - --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\": - \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }" - ``` + > Note: changes made in GitLab will be overwritten by any changes made + > upstream in Bitbucket. + + Create a file `build_status` and insert the script below and run + `chmod +x build_status` in your terminal to make the script executable. + + ```bash + #!/usr/bin/env bash + + # Push GitLab CI/CD build status to Bitbucket Cloud + + if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then + echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set" + exit 1 + fi + if [ -z "$BITBUCKET_USERNAME" ]; then + echo "ERROR: BITBUCKET_USERNAME is not set" + exit 1 + fi + if [ -z "$BITBUCKET_NAMESPACE" ]; then + echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE" + BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE + fi + if [ -z "$BITBUCKET_REPOSITORY" ]; then + echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME" + BITBUCKET_REPOSITORY=$CI_PROJECT_NAME + fi + + BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0" + BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build" + BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME" + + case "$BUILD_STATUS" in + running) + BITBUCKET_STATE="INPROGRESS" + BITBUCKET_DESCRIPTION="The build is running!" + ;; + passed) + BITBUCKET_STATE="SUCCESSFUL" + BITBUCKET_DESCRIPTION="The build passed!" + ;; + failed) + BITBUCKET_STATE="FAILED" + BITBUCKET_DESCRIPTION="The build failed." + ;; + esac + + echo "Pushing status to $BITBUCKET_STATUS_API..." + curl --request POST $BITBUCKET_STATUS_API \ + --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \ + --header "Content-Type:application/json" \ + --silent \ + --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\": + \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }" + ``` 1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push pipeline success and failures to Bitbucket. - ```yaml - stages: - - test - - ci_status - - unit-tests: - script: - - echo "Success. Add your tests!" - - success: - stage: ci_status - before_script: - - "" - after_script: - - "" - script: - - BUILD_STATUS=passed BUILD_KEY=push ./build_status - when: on_success - - failure: - stage: ci_status - before_script: - - "" - after_script: - - "" - script: - - BUILD_STATUS=failed BUILD_KEY=push ./build_status - when: on_failure - ``` + ```yaml + stages: + - test + - ci_status + + unit-tests: + script: + - echo "Success. Add your tests!" + + success: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=passed BUILD_KEY=push ./build_status + when: on_success + + failure: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=failed BUILD_KEY=push ./build_status + when: on_failure + ``` GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines configured in `.gitlab-ci.yml` and push the status to Bitbucket. diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 53b36181062..f639e3dadee 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -2,7 +2,7 @@ type: howto --- -# Using GitLab CI/CD with a GitHub repository **[PREMIUM]** +# Using GitLab CI/CD with a GitHub repository **(PREMIUM)** GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by creating a [CI/CD project](index.md) to connect your GitHub repository to @@ -19,12 +19,12 @@ administrator: 1. In GitLab create a **CI/CD for external repo** project and select **GitHub**. -  +  1. Once authenticated, you will be redirected to a list of your repositories to connect. Click **Connect** to select the repository. -  +  1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). @@ -55,14 +55,14 @@ repositories: Token**. This token with be used to access your repository and push commit statuses to GitHub. - The `repo` and `admin:repo_hook` should be enable to allow GitLab access to - your project, update commit statuses, and create a web hook to notify - GitLab of new commits. + The `repo` and `admin:repo_hook` should be enable to allow GitLab access to + your project, update commit statuses, and create a web hook to notify + GitLab of new commits. 1. In GitLab create a **CI/CD for external repo** project and select **GitHub**. -  +  1. Paste the token into the **Personal access token** field and click **List Repositories**. Click **Connect** to select the repository. @@ -86,21 +86,21 @@ your repository: Access Token.** GitLab will use this token to access your repository and push commit statuses. - Enter a **Token description** and update the scope to allow: + Enter a **Token description** and update the scope to allow: - `repo` so that GitLab can access your project and update commit statuses + `repo` so that GitLab can access your project and update commit statuses 1. In GitLab create a **CI/CD project** using the Git URL option and the HTTPS URL for your GitHub repository. If your project is private, use the personal access token you just created for authentication. - GitLab will automatically configure polling-based pull mirroring. + GitLab will automatically configure polling-based pull mirroring. 1. Still in GitLab, enable the [GitHub project integration](../../user/project/integrations/github.md) from **Settings > Integrations.** - Check the **Active** checkbox to enable the integration, paste your - personal access token and HTTPS repository URL into the form, and **Save.** + Check the **Active** checkbox to enable the integration, paste your + personal access token and HTTPS repository URL into the form, and **Save.** 1. Still in GitLab create a **Personal Access Token** with `API` scope to authenticate the GitHub web hook notifying GitLab of new commits. @@ -108,15 +108,15 @@ your repository: 1. In GitHub from **Settings > Webhooks** create a web hook to notify GitLab of new commits. - The web hook URL should be set to the GitLab API to - [trigger pull mirroring](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project-starter), - using the GitLab personal access token we just created. + The web hook URL should be set to the GitLab API to + [trigger pull mirroring](../../api/projects.md#start-the-pull-mirroring-process-for-a-project-starter), + using the GitLab personal access token we just created. - ``` - https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> - ``` + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` -  +  1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD. diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index d46e451c609..02b8eb7daa7 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -2,7 +2,7 @@ type: index, howto --- -# GitLab CI/CD for external repositories **[PREMIUM]** +# GitLab CI/CD for external repositories **(PREMIUM)** >[Introduced][ee-4642] in [GitLab Premium][eep] 10.6. diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md new file mode 100644 index 00000000000..e54be9f3bd9 --- /dev/null +++ b/doc/ci/directed_acyclic_graph/index.md @@ -0,0 +1,76 @@ +--- +type: reference +--- + +# Directed Acyclic Graph + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/47063) in GitLab 12.2 (enabled by `ci_dag_support` feature flag). + +A [directed acyclic graph](https://www.techopedia.com/definition/5739/directed-acyclic-graph-dag) can be +used in the context of a CI/CD pipeline to build relationships between jobs such that +execution is performed in the quickest possible manner, regardless how stages may +be set up. + +For example, you may have a specific tool or separate website that is built +as part of your main project. Using a DAG, you can specify the relationship between +these jobs and GitLab will then execute the jobs as soon as possible instead of waiting +for each stage to complete. + +Unlike other DAG solutions for CI/CD, GitLab does not require you to choose one or the +other. You can implement a hybrid combination of DAG and traditional +stage-based operation within a single pipeline. Configuration is kept very simple, +requiring a single keyword to enable the feature for any job. + +Consider a monorepo as follows: + +``` +./service_a +./service_b +./service_c +./service_d +``` + +It has a pipeline that looks like the following: + +| build | test | deploy | +| ----- | ---- | ------ | +| build_a | test_a | deploy_a | +| build_b | test_b | deploy_b | +| build_c | test_c | deploy_c | +| build_d | test_d | deploy_d | + +Using a DAG, you can relate the `_a` jobs to each other separately from the `_b` jobs, +and even if service `a` takes a very long time to build, service `b` will not +wait for it and will finish as quickly as it can. In this very same pipeline, `_c` and +`_d` can be left alone and will run together in staged sequence just like any normal +GitLab pipeline. + +## Use cases + +A DAG can help solve several different kinds of relationships between jobs within +a CI/CD pipeline. Most typically this would cover when jobs need to fan in or out, +and/or merge back together (diamond dependencies). This can happen when you're +handling multi-platform builds or complex webs of dependencies as in something like +an operating system build or a complex deployment graph of independently deployable +but related microservices. + +Additionally, a DAG can help with general speediness of pipelines and helping +to deliver fast feedback. By creating dependency relationships that don't unnecessarily +block each other, your pipelines will run as quickly as possible regardless of +pipeline stages, ensuring output (including errors) is available to developers +as quickly as possible. + +## Usage + +Relationships are defined between jobs using the [`needs:` keyword](../yaml/README.md#needs). + +Note that `needs:` also works with the [parallel](../yaml/README.md#parallel) keyword, +giving your powerful options for parallelization within your pipeline. + +## Limitations + +A directed acyclic graph is a complicated feature, and as of the initial MVC there +are certain use cases that you may need to work around. For more information: + +- [`needs` requirements and limitations](../yaml/README.md#requirements-and-limitations). +- Related epic [gitlab-org#1716](https://gitlab.com/groups/gitlab-org/-/epics/1716). diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index b4c4bea6447..2cbad5f101c 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -6,7 +6,6 @@ type: concepts, howto GitLab CI/CD allows you to use Docker Engine to build and test docker-based projects. - One of the new trends in Continuous Integration/Deployment is to: 1. Create an application image. @@ -29,7 +28,16 @@ during jobs. ## Runner Configuration -There are three methods to enable the use of `docker build` and `docker run` during jobs; each with their own tradeoffs. +There are three methods to enable the use of `docker build` and `docker run` +during jobs; each with their own tradeoffs. + +An alternative to using `docker build` is to [use kaniko](using_kaniko.md). +This avoids having to execute Runner in privileged mode. + +TIP: **Tip:** +To see how Docker and Runner are configured for shared Runners on +GitLab.com, see [GitLab.com Shared +Runners](../../user/gitlab_com/index.md#shared-runners). ### Use shell executor @@ -40,42 +48,42 @@ GitLab Runner then executes job scripts as the `gitlab-runner` user. 1. During GitLab Runner installation select `shell` as method of executing job scripts or use command: - ```bash - sudo gitlab-runner register -n \ - --url https://gitlab.com/ \ - --registration-token REGISTRATION_TOKEN \ - --executor shell \ - --description "My Runner" - ``` + ```bash + sudo gitlab-runner register -n \ + --url https://gitlab.com/ \ + --registration-token REGISTRATION_TOKEN \ + --executor shell \ + --description "My Runner" + ``` 1. Install Docker Engine on server. - For more information how to install Docker Engine on different systems - checkout the [Supported installations](https://docs.docker.com/engine/installation/). + For more information how to install Docker Engine on different systems + checkout the [Supported installations](https://docs.docker.com/engine/installation/). 1. Add `gitlab-runner` user to `docker` group: - ```bash - sudo usermod -aG docker gitlab-runner - ``` + ```bash + sudo usermod -aG docker gitlab-runner + ``` 1. Verify that `gitlab-runner` has access to Docker: - ```bash - sudo -u gitlab-runner -H docker info - ``` + ```bash + sudo -u gitlab-runner -H docker info + ``` - You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`: + You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`: - ```yaml - before_script: - - docker info + ```yaml + before_script: + - docker info - build_image: - script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests - ``` + build_image: + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` 1. You can now use `docker` command (and **install** `docker-compose` if needed). @@ -83,101 +91,28 @@ NOTE: **Note:** By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. For more information please read [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). -### Use docker-in-docker executor +### Use docker-in-docker workflow with Docker executor The second approach is to use the special docker-in-docker (dind) [Docker image](https://hub.docker.com/_/docker/) with all tools installed (`docker`) and run the job script in context of that -image in privileged mode. - -NOTE: **Note:** `docker-compose` is not part of docker-in-docker (dind). In case you'd like to use `docker-compose` in your CI builds, please follow the [installation instructions for docker-compose](https://docs.docker.com/compose/install/) provided by docker. - -In order to do that, follow the steps: - -1. Install [GitLab Runner](https://docs.gitlab.com/runner/install). - -1. Register GitLab Runner from the command line to use `docker` and `privileged` - mode: - - ```bash - sudo gitlab-runner register -n \ - --url https://gitlab.com/ \ - --registration-token REGISTRATION_TOKEN \ - --executor docker \ - --description "My Docker Runner" \ - --docker-image "docker:stable" \ - --docker-privileged - ``` - - The above command will register a new Runner to use the special - `docker:stable` image which is provided by Docker. **Notice that it's using - the `privileged` mode to start the build and service containers.** If you - want to use [docker-in-docker] mode, you always have to use `privileged = true` - in your Docker containers. - - The above command will create a `config.toml` entry similar to this: - - ```toml - [[runners]] - url = "https://gitlab.com/" - token = TOKEN - executor = "docker" - [runners.docker] - tls_verify = false - image = "docker:stable" - privileged = true - disable_cache = false - volumes = ["/cache"] - [runners.cache] - Insecure = false - ``` - -1. You can now use `docker` in the build script (note the inclusion of the - `docker:dind` service): +image in privileged mode. - ```yaml - image: docker:stable +NOTE: **Note:** +`docker-compose` is not part of docker-in-docker (dind). To use `docker-compose` in your +CI builds, follow the `docker-compose` +[installation instructions](https://docs.docker.com/compose/install/). - variables: - # When using dind service we need to instruct docker, to talk with the - # daemon started inside of the service. The daemon is available with - # a network connection instead of the default /var/run/docker.sock socket. - # - # The 'docker' hostname is the alias of the service container as described at - # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services - # - # Note that if you're using the Kubernetes executor, the variable should be set to - # tcp://localhost:2375/ because of how the Kubernetes executor connects services - # to the job container - # DOCKER_HOST: tcp://localhost:2375/ - # - # For non-Kubernetes executors, we use tcp://docker:2375/ - DOCKER_HOST: tcp://docker:2375/ - # When using dind, it's wise to use the overlayfs driver for - # improved performance. - DOCKER_DRIVER: overlay2 - - services: - - docker:dind - - before_script: - - docker info - - build: - stage: build - script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests - ``` +DANGER: **Danger:** +By enabling `--docker-privileged`, you are effectively disabling all of +the security mechanisms of containers and exposing your host to privilege +escalation which can lead to container breakout. For more information, check +out the official Docker documentation on +[Runtime privilege and Linux capabilities][docker-cap]. Docker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges: -- By enabling `--docker-privileged`, you are effectively disabling all of - the security mechanisms of containers and exposing your host to privilege - escalation which can lead to container breakout. For more information, check - out the official Docker documentation on - [Runtime privilege and Linux capabilities][docker-cap]. - When using docker-in-docker, each job is in a clean environment without the past history. Concurrent jobs work fine because every build gets it's own instance of Docker engine so they won't conflict with each other. But this @@ -187,7 +122,7 @@ not without its own challenges: [Using the overlayfs driver](#using-the-overlayfs-driver). - Since the `docker:dind` container and the runner container don't share their root filesystem, the job's working directory can be used as a mount point for - children containers. For example, if you have files you want to share with a + child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under `/builds/$CI_PROJECT_PATH` and use it as your mount point (for a more thorough explanation, check [issue #41227](https://gitlab.com/gitlab-org/gitlab-ce/issues/41227)): @@ -203,9 +138,187 @@ not without its own challenges: An example project using this approach can be found here: <https://gitlab.com/gitlab-examples/docker>. +In the examples below, we are using Docker images tags to specify a +specific version, such as `docker:19.03.1`. If tags like `docker:stable` +are used, you have no control over what version is going to be used and this +can lead to unpredictable behavior, especially when new versions are +released. + +#### TLS enabled + +NOTE: **Note** +This requires GitLab Runner 11.11 or higher. + +The Docker daemon supports connection over TLS and it's done by default +for Docker 19.03.1 or higher. This is the **suggested** way to use the +docker-in-docker service and +[GitLab.com Shared Runners](../../user/gitlab_com/index.html#shared-runners) +support this. + +1. Install [GitLab Runner](https://docs.gitlab.com/runner/install). + +1. Register GitLab Runner from the command line to use `docker` and `privileged` + mode: + + ```bash + sudo gitlab-runner register -n \ + --url https://gitlab.com/ \ + --registration-token REGISTRATION_TOKEN \ + --executor docker \ + --description "My Docker Runner" \ + --docker-image "docker:19.03.1" \ + --docker-privileged \ + --docker-volumes "/certs/client" + ``` + + The above command will register a new Runner to use the special + `docker:19.03.1` image, which is provided by Docker. **Notice that it's + using the `privileged` mode to start the build and service + containers.** If you want to use [docker-in-docker] mode, you always + have to use `privileged = true` in your Docker containers. + + This will also mount `/certs/client` for the service and build + container, which is needed for the docker client to use the + certificates inside of that directory. For more information how + Docker with TLS works check <https://hub.docker.com/_/docker/#tls>. + + The above command will create a `config.toml` entry similar to this: + + ```toml + [[runners]] + url = "https://gitlab.com/" + token = TOKEN + executor = "docker" + [runners.docker] + tls_verify = false + image = "docker:19.03.1" + privileged = true + disable_cache = false + volumes = ["/certs/client", "/cache"] + [runners.cache] + [runners.cache.s3] + [runners.cache.gcs] + ``` + +1. You can now use `docker` in the build script (note the inclusion of the + `docker:19.03.1-dind` service): + + ```yaml + image: docker:19.03.1 + + variables: + # When using dind service, we need to instruct docker, to talk with + # the daemon started inside of the service. The daemon is available + # with a network connection instead of the default + # /var/run/docker.sock socket. docker:19.03.1 does this automatically + # by setting the DOCKER_HOST in + # https://github.com/docker-library/docker/blob/d45051476babc297257df490d22cbd806f1b11e4/19.03.1/docker-entrypoint.sh#L23-L29 + # + # The 'docker' hostname is the alias of the service container as described at + # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services. + # + # Note that if you're using the Kubernetes executor, the variable + # should be set to tcp://localhost:2376/ because of how the + # Kubernetes executor connects services to the job container + # DOCKER_HOST: tcp://localhost:2376/ + # + # When using dind, it's wise to use the overlayfs driver for + # improved performance. + DOCKER_DRIVER: overlay2 + # Specify to Docker where to create the certificates, Docker will + # create them automatically on boot, and will create + # `/certs/client` that will be shared between the service and job + # container, thanks to volume mount from config.toml + DOCKER_TLS_CERTDIR: "/certs" + + services: + - docker:19.03.1-dind + + before_script: + - docker info + + build: + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +#### TLS disabled + +Sometimes there are legitimate reasons why you might want to disable TLS. +For example, you have no control over the GitLab Runner configuration +that you are using. + +Assuming that the Runner `config.toml` is similar to: + +```toml +[[runners]] + url = "https://gitlab.com/" + token = TOKEN + executor = "docker" + [runners.docker] + tls_verify = false + image = "docker:19.03.1" + privileged = true + disable_cache = false + volumes = ["/cache"] + [runners.cache] + [runners.cache.s3] + [runners.cache.gcs] +``` + +You can now use `docker` in the build script (note the inclusion of the +`docker:19.03.1-dind` service): + +```yaml +image: docker:19.03.1 + +variables: + # When using dind service we need to instruct docker, to talk with the + # daemon started inside of the service. The daemon is available with + # a network connection instead of the default /var/run/docker.sock socket. + # + # The 'docker' hostname is the alias of the service container as described at + # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services + # + # Note that if you're using the Kubernetes executor, the variable should be set to + # tcp://localhost:2375/ because of how the Kubernetes executor connects services + # to the job container + # DOCKER_HOST: tcp://localhost:2375/ + # + # For non-Kubernetes executors, we use tcp://docker:2375/ + DOCKER_HOST: tcp://docker:2375/ + # When using dind, it's wise to use the overlayfs driver for + # improved performance. + DOCKER_DRIVER: overlay2 + # + # This will instruct Docker not to start over TLS. + DOCKER_TLS_CERTDIR: "" + +services: + - docker:19.03.1-dind + +before_script: + - docker info + +build: + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests +``` + ### Use Docker socket binding -The third approach is to bind-mount `/var/run/docker.sock` into the container so that docker is available in the context of that image. +The third approach is to bind-mount `/var/run/docker.sock` into the +container so that Docker is available in the context of that image. + +NOTE: **Note:** +If you bind the Docker socket [when using GitLab Runner 11.11 or +newer](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1261), +you can no longer use `docker:dind` as a service because volume bindings +are done to the services as well, making these incompatible. In order to do that, follow the steps: @@ -213,54 +326,54 @@ In order to do that, follow the steps: 1. Register GitLab Runner from the command line to use `docker` and share `/var/run/docker.sock`: - ```bash - sudo gitlab-runner register -n \ - --url https://gitlab.com/ \ - --registration-token REGISTRATION_TOKEN \ - --executor docker \ - --description "My Docker Runner" \ - --docker-image "docker:stable" \ - --docker-volumes /var/run/docker.sock:/var/run/docker.sock - ``` - - The above command will register a new Runner to use the special - `docker:stable` image which is provided by Docker. **Notice that it's using - the Docker daemon of the Runner itself, and any containers spawned by docker - commands will be siblings of the Runner rather than children of the runner.** - This may have complications and limitations that are unsuitable for your workflow. - - The above command will create a `config.toml` entry similar to this: - - ```toml - [[runners]] - url = "https://gitlab.com/" - token = REGISTRATION_TOKEN - executor = "docker" - [runners.docker] - tls_verify = false - image = "docker:stable" - privileged = false - disable_cache = false - volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] - [runners.cache] - Insecure = false - ``` + ```bash + sudo gitlab-runner register -n \ + --url https://gitlab.com/ \ + --registration-token REGISTRATION_TOKEN \ + --executor docker \ + --description "My Docker Runner" \ + --docker-image "docker:stable" \ + --docker-volumes /var/run/docker.sock:/var/run/docker.sock + ``` + + The above command will register a new Runner to use the special + `docker:stable` image which is provided by Docker. **Notice that it's using + the Docker daemon of the Runner itself, and any containers spawned by docker + commands will be siblings of the Runner rather than children of the runner.** + This may have complications and limitations that are unsuitable for your workflow. + + The above command will create a `config.toml` entry similar to this: + + ```toml + [[runners]] + url = "https://gitlab.com/" + token = REGISTRATION_TOKEN + executor = "docker" + [runners.docker] + tls_verify = false + image = "docker:stable" + privileged = false + disable_cache = false + volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] + [runners.cache] + Insecure = false + ``` 1. You can now use `docker` in the build script (note that you don't need to include the `docker:dind` service as when using the Docker in Docker executor): - ```yaml - image: docker:stable + ```yaml + image: docker:stable - before_script: - - docker info + before_script: + - docker info - build: - stage: build - script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests - ``` + build: + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` While the above method avoids using Docker in privileged mode, you should be aware of the following implications: @@ -276,9 +389,9 @@ aware of the following implications: work as expected since volume mounting is done in the context of the host machine, not the build container. For example: - ```sh - docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests - ``` + ```sh + docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests + ``` ## Making docker-in-docker builds faster with Docker layer caching @@ -349,23 +462,23 @@ which can be avoided if a different driver is used, for example `overlay2`. 1. Make sure a recent kernel is used, preferably `>= 4.2`. 1. Check whether the `overlay` module is loaded: - ```sh - sudo lsmod | grep overlay - ``` + ```sh + sudo lsmod | grep overlay + ``` - If you see no result, then it isn't loaded. To load it use: + If you see no result, then it isn't loaded. To load it use: - ```sh - sudo modprobe overlay - ``` + ```sh + sudo modprobe overlay + ``` - If everything went fine, you need to make sure module is loaded on reboot. - On Ubuntu systems, this is done by editing `/etc/modules`. Just add the - following line into it: + If everything went fine, you need to make sure module is loaded on reboot. + On Ubuntu systems, this is done by editing `/etc/modules`. Just add the + following line into it: - ```text - overlay - ``` + ```text + overlay + ``` ### Use driver per project @@ -433,9 +546,9 @@ For all projects, mostly suitable for public ones: your Docker images and has read/write access to the Registry. This is ephemeral, so it's only valid for one job. You can use the following example as-is: - ```sh - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY - ``` + ```sh + docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY + ``` For private and internal projects: @@ -448,9 +561,9 @@ For private and internal projects: Replace the `<username>` and `<access_token>` in the following example: - ```sh - docker login -u <username> -p <access_token> $CI_REGISTRY - ``` + ```sh + docker login -u <username> -p <access_token> $CI_REGISTRY + ``` - **Using the GitLab Deploy Token**: You can create and use a [special deploy token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token) @@ -458,9 +571,9 @@ For private and internal projects: Once created, you can use the special environment variables, and GitLab CI/CD will fill them in for you. You can use the following example as-is: - ```sh - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY - ``` + ```sh + docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY + ``` ### Container Registry examples diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 29578efacbb..489791141ed 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -35,8 +35,8 @@ sudo gitlab-runner register \ --description "docker-ruby-2.1" \ --executor "docker" \ --docker-image ruby:2.1 \ - --docker-postgres latest \ - --docker-mysql latest + --docker-services postgres:latest \ + --docker-services mysql:latest ``` The registered runner will use the `ruby:2.1` Docker image and will run two @@ -193,13 +193,14 @@ You can simply define an image that will be used for all jobs and a list of services that you want to use during build time: ```yaml -image: ruby:2.2 +default: + image: ruby:2.2 -services: - - postgres:9.3 + services: + - postgres:9.3 -before_script: - - bundle install + before_script: + - bundle install test: script: @@ -209,8 +210,9 @@ test: It is also possible to define different images and services per job: ```yaml -before_script: - - bundle install +default: + before_script: + - bundle install test:2.1: image: ruby:2.1 @@ -231,16 +233,53 @@ Or you can pass some [extended configuration options](#extended-docker-configura for `image` and `services`: ```yaml +default: + image: + name: ruby:2.2 + entrypoint: ["/bin/bash"] + + services: + - name: my-postgres:9.4 + alias: db-postgres + entrypoint: ["/usr/local/bin/db-postgres"] + command: ["start"] + + before_script: + - bundle install + +test: + script: + - bundle exec rake spec +``` + +## Passing environment variables to services + +You can also pass custom environment [variables](../variables/README.md) +to fine tune your Docker `images` and `services` directly in the `.gitlab-ci.yml` file. +For more information, see [custom environment variables](../variables/README.md#gitlab-ciyml-defined-variables) + +```yaml +# The following variables will automatically be passed down to the Postgres container +# as well as the Ruby container and available within each. +variables: + HTTPS_PROXY: "https://10.1.1.1:8090" + HTTP_PROXY: "https://10.1.1.1:8090" + POSTGRES_DB: "my_custom_db" + POSTGRES_USER: "postgres" + POSTGRES_PASSWORD: "example" + PGDATA: "/var/lib/postgresql/data" + POSTGRES_INITDB_ARGS: "--encoding=UTF8 --data-checksums" + +services: +- name: postgres:9.4 + alias: db + entrypoint: ["docker-entrypoint.sh"] + command: ["postgres"] + image: name: ruby:2.2 entrypoint: ["/bin/bash"] -services: -- name: my-postgres:9.4 - alias: db-postgres - entrypoint: ["/usr/local/bin/db-postgres"] - command: ["start"] - before_script: - bundle install @@ -266,25 +305,25 @@ For example, the following two definitions are equal: 1. Using a string as an option to `image` and `services`: - ```yaml - image: "registry.example.com/my/image:latest" + ```yaml + image: "registry.example.com/my/image:latest" - services: - - postgresql:9.4 - - redis:latest - ``` + services: + - postgresql:9.4 + - redis:latest + ``` 1. Using a map as an option to `image` and `services`. The use of `image:name` is required: - ```yaml - image: - name: "registry.example.com/my/image:latest" + ```yaml + image: + name: "registry.example.com/my/image:latest" - services: - - name: postgresql:9.4 - - name: redis:latest - ``` + services: + - name: postgresql:9.4 + - name: redis:latest + ``` ### Available settings for `image` @@ -456,16 +495,6 @@ that runner. ## Define an image from a private Container Registry -> **Notes:** -> -> - This feature requires GitLab Runner **1.8** or higher -> - For GitLab Runner versions **>= 0.6, <1.8** there was a partial -> support for using private registries, which required manual configuration -> of credentials on runner's host. We recommend to upgrade your Runner to -> at least version **1.8** if you want to use private registries. -> - If the repository is private you need to authenticate your GitLab Runner in the -> registry. Learn more about how [GitLab Runner works in this case][runner-priv-reg]. - To access private container registries, the GitLab Runner process can use: - [Statically defined credentials](#using-statically-defined-credentials). That is, a username and password for a specific registry. @@ -478,17 +507,43 @@ To define which should be used, the GitLab Runner process reads the configuratio - A [variable](../variables/README.md#gitlab-cicd-environment-variables) in `.gitlab-ci.yml`. - A project's variables stored on the projects **Settings > CI/CD** page. - `DOCKER_AUTH_CONFIG` variable provided as environment variable in `config.toml` of the Runner. -- `config.json` file placed in `$HOME/docker` directory of the user running GitLab Runner process. +- `config.json` file placed in `$HOME/.docker` directory of the user running GitLab Runner process. If the `--user` flag is provided to run the GitLab Runner child processes as unprivileged user, the home directory of the main GitLab Runner process user will be used. -NOTE: **Note:** -GitLab Runner reads this configuration **only** from `config.toml` and ignores it if +NOTE: **Note:** +GitLab Runner reads this configuration **only** from `config.toml` and ignores it if it's provided as an environment variable. This is because GitLab Runnner uses **only** `config.toml` configuration and doesn't interpolate **ANY** environment variables at runtime. +### Requirements and limitations + +- This feature requires GitLab Runner **1.8** or higher. +- For GitLab Runner versions **>= 0.6, <1.8** there was a partial + support for using private registries, which required manual configuration + of credentials on runner's host. We recommend to upgrade your Runner to + at least version **1.8** if you want to use private registries. +- Not available for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html), + follow <https://gitlab.com/gitlab-org/gitlab-runner/issues/2673> for + details. + ### Using statically-defined credentials + +There are two approaches that you can take in order to access a +private registry. Both require setting the environment variable +`DOCKER_AUTH_CONFIG` with appropriate authentication info. + +1. Per-job: To configure one job to access a private registry, add + `DOCKER_AUTH_CONFIG` as a job variable. +1. Per-runner: To configure a Runner so all its jobs can access a + private registry, add `DOCKER_AUTH_CONFIG` to the environment in the + Runner's configuration. + +See below for examples of each. + +#### Determining your `DOCKER_AUTH_CONFIG` data + As an example, let's assume that you want to use the `registry.example.com:5000/private/image:latest` image which is private and requires you to login into a private container registry. @@ -500,61 +555,64 @@ Let's also assume that these are the login credentials: | username | `my_username` | | password | `my_password` | -To configure access for `registry.example.com:5000`, follow these steps: +There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: -1. Find what the value of `DOCKER_AUTH_CONFIG` should be. There are two ways to - accomplish this: - - **First way -** Do a `docker login` on your local machine: +- **First way -** Do a `docker login` on your local machine: - ```bash - docker login registry.example.com:5000 --username my_username --password my_password - ``` + ```bash + docker login registry.example.com:5000 --username my_username --password my_password + ``` - Then copy the content of `~/.docker/config.json`. - - **Second way -** In some setups, it's possible that Docker client will use - the available system keystore to store the result of `docker login`. In - that case, it's impossible to read `~/.docker/config.json`, so you will - need to prepare the required base64-encoded version of - `${username}:${password}` manually. Open a terminal and execute the - following command: + Then copy the content of `~/.docker/config.json`. - ```bash - echo -n "my_username:my_password" | base64 + If you don't need access to the registry from your computer, you + can do a `docker logout`: - # Example output to copy - bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= - ``` + ```bash + docker logout registry.example.com:5000 + ``` -1. Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the - Docker configuration file as the value: +- **Second way -** In some setups, it's possible that Docker client + will use the available system keystore to store the result of `docker + login`. In that case, it's impossible to read `~/.docker/config.json`, + so you will need to prepare the required base64-encoded version of + `${username}:${password}` manually. Open a terminal and execute the + following command: + + ```bash + echo -n "my_username:my_password" | base64 - ```json - { - "auths": { - "registry.example.com:5000": { - "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=" - } - } - } - ``` + # Example output to copy + bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= + ``` -1. Optionally,if you followed the first way of finding the `DOCKER_AUTH_CONFIG` - value, do a `docker logout` on your computer if you don't need access to the - registry from it: +#### Configuring a job - ```bash - docker logout registry.example.com:5000 - ``` +To configure a single job with access for `registry.example.com:5000`, +follow these steps: + +1. Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the + Docker configuration file as the value: + + ```json + { + "auths": { + "registry.example.com:5000": { + "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=" + } + } + } + ``` 1. You can now use any private image from `registry.example.com:5000` defined in `image` and/or `services` in your `.gitlab-ci.yml` file: - ```yaml - image: registry.example.com:5000/namespace/image:tag - ``` + ```yaml + image: registry.example.com:5000/namespace/image:tag + ``` - In the example above, GitLab Runner will look at `registry.example.com:5000` for the - image `namespace/image:tag`. + In the example above, GitLab Runner will look at `registry.example.com:5000` for the + image `namespace/image:tag`. You can add configuration for as many registries as you want, adding more registries to the `"auths"` hash as described above. @@ -566,6 +624,37 @@ for the Runner to match the `DOCKER_AUTH_CONFIG`. For example, if then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`. Specifying only `registry.example.com` will not work. +### Configuring a Runner + +If you have many pipelines that access the same registry, it'll +probably be better to setup registry access at the runner level. This +allows pipeline authors to have access to a private registry just by +running a job on the appropriate runner. It also makes registry +changes and credential rotations much simpler. + +Of course this means that any job on that runner can access the +registry with the same privilege, even across projects. If you need to +control access to the registry, you'll need to be sure to control +access to the runner. + +To add `DOCKER_AUTH_CONFIG` to a Runner: + +1. Modify the Runner's `config.toml` file as follows: + + ```toml + [[runners]] + environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.example.com:5000\":{\"auth\":\"bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=\"}}}"] + ``` + +1. Restart the Runner service. + +NOTE: **Note:** The double quotes included in the `DOCKER_AUTH_CONFIG` +data must be escaped with backslashes. This prevents them from being +interpreted as TOML. + +NOTE: **Note:** The `environment` option is a list. So your Runner may +have existing entries and you should add this to the list, not replace +it. ### Using Credentials Store @@ -574,19 +663,20 @@ Specifying only `registry.example.com` will not work. To configure credentials store, follow these steps: 1. To use a credentials store, you need an external helper program to interact with a specific keychain or external store. -Make sure helper program is available in GitLab Runner `$PATH`. + Make sure helper program is available in GitLab Runner `$PATH`. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: - - Create a + + - Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the - Docker configuration file as the value: + Docker configuration file as the value: - ```json - { - "credsStore": "osxkeychain" - } - ``` + ```json + { + "credsStore": "osxkeychain" + } + ``` - Or, if you are running self-hosted Runners, add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner will read this config file @@ -608,17 +698,18 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th 1. Make sure `docker-credential-ecr-login` is available in GitLab Runner's `$PATH`. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: + - Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the - Docker configuration file as the value: + Docker configuration file as the value: - ```json - { - "credHelpers": { - "aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login" - } - } - ``` + ```json + { + "credHelpers": { + "aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login" + } + } + ``` - Or, if you are running self-hosted Runners, add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. @@ -628,12 +719,12 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th 1. You can now use any private image from `aws_account_id.dkr.ecr.region.amazonaws.com` defined in `image` and/or `services` in your `.gitlab-ci.yml` file: - ```yaml - image: aws_account_id.dkr.ecr.region.amazonaws.com/private/image:latest - ``` + ```yaml + image: aws_account_id.dkr.ecr.region.amazonaws.com/private/image:latest + ``` - In the example above, GitLab Runner will look at `aws_account_id.dkr.ecr.region.amazonaws.com` for the - image `private/image:latest`. + In the example above, GitLab Runner will look at `aws_account_id.dkr.ecr.region.amazonaws.com` for the + image `private/image:latest`. You can add configuration for as many registries as you want, adding more registries to the `"credHelpers"` hash as described above. @@ -741,7 +832,6 @@ creation. [tutum/wordpress]: https://hub.docker.com/r/tutum/wordpress/ [postgres-hub]: https://hub.docker.com/r/_/postgres/ [mysql-hub]: https://hub.docker.com/r/_/mysql/ -[runner-priv-reg]: http://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry [entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint [cmd]: https://docs.docker.com/engine/reference/builder/#cmd [register]: https://docs.gitlab.com/runner/register/ diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 50f1ac3d54a..925653f9fdf 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -11,7 +11,8 @@ Requires GitLab Runner 11.2 and above. container images from a Dockerfile, inside a container or Kubernetes cluster. kaniko solves two problems with using the -[docker-in-docker build](using_docker_build.md#use-docker-in-docker-executor) method: +[docker-in-docker +build](using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor) method: - Docker-in-docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) in order to function, which is a significant security concern. diff --git a/doc/ci/environments.md b/doc/ci/environments.md index a32dbc11a33..f6c47a99712 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -591,15 +591,13 @@ exist, you should see something like: ### Monitoring environments -> **Notes:** -> -> - For the monitoring dashboard to appear, you need to: -> - Enable the [Prometheus integration](../user/project/integrations/prometheus.md). -> - Configure Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/index.md). -> - With GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. - If you have enabled [Prometheus for monitoring system and response metrics](../user/project/integrations/prometheus.md), -you can monitor the behavior of your app running in each environment. +you can monitor the behavior of your app running in each environment. For the monitoring +dashboard to appear, you need to Configure Prometheus to collect at least one +[supported metric](../user/project/integrations/prometheus_library/index.md). + +NOTE: **Note:** +Since GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. Once configured, GitLab will attempt to retrieve [supported performance metrics](../user/project/integrations/prometheus_library/index.md) for any environment that has had a successful deployment. If monitoring data was @@ -621,6 +619,10 @@ versions of the app, all without leaving GitLab. Add a [button to the Monitoring dashboard](../user/project/operations/linking_to_an_external_dashboard.md) linking directly to your existing external dashboards. +#### Embedding metrics in GitLab Flavored Markdown + +Metric charts can be embedded within GitLab Flavored Markdown. See [Embedding Metrics within GitLab Flavored Markdown](../user/project/integrations/prometheus.md#embedding-metric-charts-within-gitlab-flavored-markdown) for more details. + ### Web terminals > Web terminals were added in GitLab 8.15 and are only available to project Maintainers and Owners. @@ -673,7 +675,7 @@ fetch line: fetch = +refs/environments/*:refs/remotes/origin/environments/* ``` -### Scoping environments with specs **[PREMIUM]** +### Scoping environments with specs **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2112) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. @@ -692,7 +694,7 @@ with `review/` would have that particular variable. Some GitLab features can behave differently for each environment. For example, you can -[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables-premium). **[PREMIUM]** +[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables). In most cases, these features use the _environment specs_ mechanism, which offers an efficient way to implement scoping within each environment group. @@ -734,7 +736,7 @@ Below are some links you may find interesting: - [The `.gitlab-ci.yml` definition of environments](yaml/README.md#environment) - [A blog post on Deployments & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) - [Review Apps - Use dynamic environments to deploy your code for every branch](review_apps/index.md) -- [Deploy Boards for your applications running on Kubernetes](../user/project/deploy_boards.md) **[PREMIUM]** +- [Deploy Boards for your applications running on Kubernetes](../user/project/deploy_boards.md) **(PREMIUM)** <!-- ## Troubleshooting diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index b72ebe838b8..e5213881862 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -2,7 +2,7 @@ type: concepts, howto --- -# Protected Environments **[PREMIUM]** +# Protected Environments **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6303) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 2b4fe321cb3..00995f881da 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -33,19 +33,20 @@ The following table lists examples with step-by-step tutorials that are containe | Java with Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | | PHP with PHPunit, atoum | [Testing PHP projects](php.md). | | PHP with NPM, SCP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | -| PHP with Laravel, Ennvoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | +| PHP with Laravel, Envoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | | Python on Heroku | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | | Ruby on Heroku | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | | Scala on Heroku | [Test and deploy a Scala application to Heroku](test-scala-application.md). | +| Parallel testing Ruby & JS | [GitLab CI parallel jobs testing for Ruby & JavaScript projects](https://docs.knapsackpro.com/2019/how-to-run-parallel-jobs-for-rspec-tests-on-gitlab-ci-pipeline-and-speed-up-ruby-javascript-testing). | ### Contributing examples Contributions are welcome! You can help your favorite programming language users and GitLab by sending a merge request with a guide for that language. -You may want to apply for the [GitLab Community Writers Program](https://about.gitlab.com/community-writers/) +You may want to apply for the [GitLab Community Writers Program](https://about.gitlab.com/community/writers/) to get paid for writing complete articles for GitLab. -## Adding templates to your GitLab installation **[PREMIUM ONLY]** +## Adding templates to your GitLab installation **(PREMIUM ONLY)** If you want to have customized examples and templates for your own self-managed GitLab instance available to your team, your GitLab administrator can [designate an instance template repository](../../user/admin_area/settings/instance_template_repository.md) that contains examples and templates specific to your enterprise. diff --git a/doc/ci/examples/artifactory_and_gitlab/index.md b/doc/ci/examples/artifactory_and_gitlab/index.md index 2117b342903..e5f307e570d 100644 --- a/doc/ci/examples/artifactory_and_gitlab/index.md +++ b/doc/ci/examples/artifactory_and_gitlab/index.md @@ -1,5 +1,5 @@ --- -redirect_from: 'https://docs.gitlab.com/ee/articles/artifactory_and_gitlab/index.html' +disqus_identifier: 'https://docs.gitlab.com/ee/articles/artifactory_and_gitlab/index.html' author: Fabio Busatto author_gitlab: bikebilly level: intermediate @@ -13,7 +13,7 @@ date: 2017-08-15 ## Introduction In this article, we will show how you can leverage the power of [GitLab CI/CD](https://about.gitlab.com/product/continuous-integration/) -to build a [Maven](https://maven.apache.org/) project, deploy it to [Artifactory](https://www.jfrog.com/artifactory/), and then use it from another Maven application as a dependency. +to build a [Maven](https://maven.apache.org/) project, deploy it to [Artifactory](https://jfrog.com/artifactory/), and then use it from another Maven application as a dependency. You'll create two different projects: @@ -39,9 +39,10 @@ project: 1. Create a new project by selecting **Import project from ➔ Repo by URL** 1. Add the following URL: - ``` - https://gitlab.com/gitlab-examples/maven/simple-maven-dep.git - ``` + ``` + https://gitlab.com/gitlab-examples/maven/simple-maven-dep.git + ``` + 1. Click **Create project** This application is nothing more than a basic class with a stub for a JUnit based test suite. @@ -63,15 +64,15 @@ The application is ready to use, but you need some additional steps to deploy it 1. Copy the snippet in the `pom.xml` file for your project, just after the `dependencies` section. The snippet should look like this: - ```xml - <distributionManagement> - <repository> - <id>central</id> - <name>83d43b5afeb5-releases</name> - <url>${env.MAVEN_REPO_URL}/libs-release-local</url> - </repository> - </distributionManagement> - ``` + ```xml + <distributionManagement> + <repository> + <id>central</id> + <name>83d43b5afeb5-releases</name> + <url>${env.MAVEN_REPO_URL}/libs-release-local</url> + </repository> + </distributionManagement> + ``` Another step you need to do before you can deploy the dependency to Artifactory is to configure the authentication data. It is a simple task, but Maven requires @@ -86,18 +87,18 @@ parameter in `.gitlab-ci.yml` to use the custom location instead of the default 1. Create a file called `settings.xml` in the `.m2` folder 1. Copy the following content into a `settings.xml` file: - ```xml - <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" - xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> - <servers> - <server> - <id>central</id> - <username>${env.MAVEN_REPO_USER}</username> - <password>${env.MAVEN_REPO_PASS}</password> - </server> - </servers> - </settings> - ``` + ```xml + <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" + xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> + <servers> + <server> + <id>central</id> + <username>${env.MAVEN_REPO_USER}</username> + <password>${env.MAVEN_REPO_PASS}</password> + </server> + </servers> + </settings> + ``` Username and password will be replaced by the correct values using variables. @@ -187,9 +188,10 @@ We'll use again a Maven app that can be cloned from our example project: 1. Create a new project by selecting **Import project from ➔ Repo by URL** 1. Add the following URL: - ``` - https://gitlab.com/gitlab-examples/maven/simple-maven-app.git - ``` + ``` + https://gitlab.com/gitlab-examples/maven/simple-maven-app.git + ``` + 1. Click **Create project** This one is a simple app as well. If you look at the `src/main/java/com/example/app/App.java` @@ -204,13 +206,13 @@ Since Maven doesn't know how to resolve the dependency, you need to modify the c 1. Copy the snippet in the `dependencies` section of the `pom.xml` file. The snippet should look like this: - ```xml - <dependency> - <groupId>com.example.dep</groupId> - <artifactId>simple-maven-dep</artifactId> - <version>1.0</version> - </dependency> - ``` + ```xml + <dependency> + <groupId>com.example.dep</groupId> + <artifactId>simple-maven-dep</artifactId> + <version>1.0</version> + </dependency> + ``` ### Configure the Artifactory repository location diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index 8ecac4a5a4f..3266e5dc62e 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -15,7 +15,7 @@ your code by using GitLab CI/CD and [sitespeed.io](https://www.sitespeed.io) using Docker-in-Docker. First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). +[docker-in-docker build](../docker/using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor). Once you set up the Runner, add a new job to `.gitlab-ci.yml` that generates the expected report: @@ -155,4 +155,4 @@ performance: paths: - performance.json - sitespeed-results/ -```
\ No newline at end of file +``` diff --git a/doc/ci/examples/code_quality.md b/doc/ci/examples/code_quality.md index 43f773dab7c..9c65de115b4 100644 --- a/doc/ci/examples/code_quality.md +++ b/doc/ci/examples/code_quality.md @@ -1,5 +1,5 @@ --- -redirect_from: 'https://docs.gitlab.com/ee/ci/examples/code_climate.html' +disqus_identifier: 'https://docs.gitlab.com/ee/ci/examples/code_climate.html' type: reference, howto --- @@ -14,7 +14,7 @@ This example shows how to run Code Quality on your code by using GitLab CI/CD and Docker. First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). +[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor). Once you set up the Runner, include the CodeQuality template in your CI config: @@ -34,6 +34,12 @@ For [GitLab Starter][ee] users, this information will be automatically extracted and shown right in the merge request widget. [Learn more on Code Quality in merge requests](../../user/project/merge_requests/code_quality.md). +CAUTION: **Caution:** +On self-managed instances, if a malicious actor compromises the Code Quality job +definition they will be able to execute privileged docker commands on the Runner +host. Having proper access control policies mitigates this attack vector by +allowing access only to trusted actors. + ## Previous job definitions CAUTION: **Caution:** diff --git a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md index 538843ab8dc..1d4c9221cf2 100644 --- a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md +++ b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md @@ -79,10 +79,10 @@ image: java:8 stages: - build - deploy - + before_script: - chmod +x mvnw - + build: stage: build script: ./mvnw package diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md index 50e61cafeb9..44d3ec8046c 100644 --- a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -418,10 +418,14 @@ fully understand [IAM Best Practices in AWS](https://docs.aws.amazon.com/IAM/lat 1. Log into your AWS account and go to the [Security Credentials page](https://console.aws.amazon.com/iam/home#/security_credential) 1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the id and secret around, you'll need them later -  + +  + 1. Go to your GitLab project, click **Settings > CI/CD** on the left sidebar 1. Expand the **Variables** section -  + +  + 1. Add a key named `AWS_KEY_ID` and copy the key id from Step 2 into the **Value** textbox 1. Add a key named `AWS_KEY_SECRET` and copy the key secret from Step 2 into the **Value** textbox diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/img/deployed_dependency_update.png b/doc/ci/examples/end_to_end_testing_webdriverio/img/deployed_dependency_update.png Binary files differindex c45d70d7f7a..9fe1739f37e 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/img/deployed_dependency_update.png +++ b/doc/ci/examples/end_to_end_testing_webdriverio/img/deployed_dependency_update.png diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 7f1beb96bbf..38d0d86ffa2 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -32,7 +32,7 @@ through the process of setting up GitLab CI/CD for end-to-end testing Javascript with WebdriverIO, but the general strategy should carry over to other languages. We assume you are familiar with GitLab, [GitLab CI/CD](../../README.md), [Review Apps](../../review_apps/index.md), and running your app locally, e.g., on `localhost:8000`. -### What to test +## What to test In the widely-used [testing pyramid strategy](https://martinfowler.com/bliki/TestPyramid.html), end-to-end tests act more like a safeguard: [most of your code should be covered by @@ -40,9 +40,9 @@ unit tests](https://vincenttunru.com/100-percent-coverage/) that allow you to ea will likely want to [limit the number of end-to-end tests](https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html) to just enough to give you the confidence that the deployment went as intended, that your -infrastructure is up and running, and that your units of code work well together. +infrastructure is up and running, and that your units of code work well together. -### Selenium and WebdriverIO +## Selenium and WebdriverIO [Selenium](http://www.seleniumhq.org/) is a piece of software that can control web browsers, e.g., to make them visit a specific URL or interact with elements on the page. It can be programmatically controlled @@ -65,7 +65,7 @@ describe('A visitor without account', function(){ expect(browser.getUrl()).toMatch('page-that-does-not-exist'); browser.element('.content a[href="/"]').click(); - + expect(browser.getUrl()).not.toMatch('page-that-does-not-exist'); }); }); @@ -143,7 +143,7 @@ Which brings us to the exciting part: how do we run this in GitLab CI/CD? There need to do for this: 1. Set up [CI/CD jobs](../../yaml/README.md#introduction) that actually have a browser available. -2. Update our WebdriverIO configuration to use those browsers to visit the review apps. +1. Update our WebdriverIO configuration to use those browsers to visit the review apps. For the scope of this article, we've defined an additional [CI/CD stage](../../yaml/README.md#stages) `confidence-check` that is executed _after_ the stage that deploys the review app. It uses the `node:latest` [Docker diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index d7308a3a5ec..808e4285f2f 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -1,5 +1,5 @@ --- -redirect_from: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' +disqus_identifier: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' author: Mehran Rasulian author_gitlab: mehranrasulian level: intermediate @@ -452,6 +452,7 @@ To start using Container Registry on our machine, we first need to login to the ```bash docker login registry.gitlab.com ``` + Then we can build and push our image to GitLab: ```bash diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index c459bb7001f..1dd3049d53d 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -6,7 +6,7 @@ type: tutorial This guide covers basic building instructions for PHP projects. -Two testing scenarios are covered: using the Docker executor and +Two testing scenarios are covered: using the Docker executor and using the Shell executor. ## Test PHP projects using the Docker executor @@ -64,7 +64,7 @@ docker-php-ext-install pdo_mysql You might wonder what `docker-php-ext-install` is. In short, it is a script provided by the official php docker image that you can use to easily install extensions. For more information read the documentation at -<https://hub.docker.com/r/_/php/>. +<https://hub.docker.com/_/php>. Now that we created the script that contains all prerequisites for our build environment, let's add it in `.gitlab-ci.yml`: @@ -96,7 +96,7 @@ Finally, commit your files and push them to GitLab to see your build succeeding The final `.gitlab-ci.yml` should look similar to this: ```yaml -# Select image from https://hub.docker.com/r/_/php/ +# Select image from https://hub.docker.com/_/php image: php:5.6 before_script: @@ -286,7 +286,7 @@ that runs on [GitLab.com](https://gitlab.com) using our publicly available Want to hack on it? Simply fork it, commit, and push your changes. Within a few moments the changes will be picked by a public runner and the job will begin. -[php-hub]: https://hub.docker.com/r/_/php/ +[php-hub]: https://hub.docker.com/_/php [phpenv]: https://github.com/phpenv/phpenv [phpenv-installation]: https://github.com/phpenv/phpenv#installation [php-example-repo]: https://gitlab.com/gitlab-examples/php diff --git a/doc/ci/examples/test-clojure-application.md b/doc/ci/examples/test-clojure-application.md index 5cda8702b56..6ea38f22bca 100644 --- a/doc/ci/examples/test-clojure-application.md +++ b/doc/ci/examples/test-clojure-application.md @@ -35,7 +35,7 @@ test: - lein test ``` -In `before_script`, we install JRE and [Leiningen](http://leiningen.org/). +In `before_script`, we install JRE and [Leiningen](https://leiningen.org/). The sample project uses the [migratus](https://github.com/yogthos/migratus) library to manage database migrations, and we have added a database migration as the last step of `before_script`. diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md index 0e33a1ba060..7d039ab1aeb 100644 --- a/doc/ci/examples/test-scala-application.md +++ b/doc/ci/examples/test-scala-application.md @@ -46,11 +46,11 @@ deploy: In the above configuration: -- The `before_script` installs [SBT](http://www.scala-sbt.org/) and -displays the version that is being used. +- The `before_script` installs [SBT](https://www.scala-sbt.org/) and + displays the version that is being used. - The `test` stage executes SBT to compile and test the project. - - [sbt-scoverage](https://github.com/scoverage/sbt-scoverage) is used as an SBT -plugin to measure test coverage. + - [sbt-scoverage](https://github.com/scoverage/sbt-scoverage) is used as an SBT + plugin to measure test coverage. - The `deploy` stage automatically deploys the project to Heroku using dpl. You can use other versions of Scala and SBT by defining them in diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index ec25ca1bfc3..0e595e1a0be 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -188,28 +188,27 @@ when running our Phoenix in our `localhost`. - Open `hello_gitlab_ci/config/test.exs` on your favorite code editor - Go to **Configure your database** session and edit the block to include `System.get_env`: - ```elixir - # Configure your database - config :hello_gitlab_ci, HelloGitlabCi.Repo, - adapter: Ecto.Adapters.Postgres, - username: System.get_env("POSTGRES_USER") || "postgres", - password: System.get_env("POSTGRES_PASSWORD") || "postgres", - database: System.get_env("POSTGRES_DB") || "hello_gitlab_ci_test", - hostname: System.get_env("POSTGRES_HOST") || "localhost", - pool: Ecto.Adapters.SQL.Sandbox - ``` - - We'll need these system variables later on. + ```elixir + # Configure your database + config :hello_gitlab_ci, HelloGitlabCi.Repo, + adapter: Ecto.Adapters.Postgres, + username: System.get_env("POSTGRES_USER") || "postgres", + password: System.get_env("POSTGRES_PASSWORD") || "postgres", + database: System.get_env("POSTGRES_DB") || "hello_gitlab_ci_test", + hostname: System.get_env("POSTGRES_HOST") || "localhost", + pool: Ecto.Adapters.SQL.Sandbox + ``` + + We'll need these system variables later on. - Create an empty file named `.gitkeep` into `hello_gitlab_ci/priv/repo/migrations` - As our project is still fresh, we don't have any data on our database, so, the `migrations` -directory will be empty. - Without `.gitkeep`, git will not upload this empty directory and we'll got an error when running our -test on GitLab. + As our project is still fresh, we don't have any data on our database, so, the `migrations` + directory will be empty. + Without `.gitkeep`, git will not upload this empty directory and we'll got an error when running our + test on GitLab. - > **Note:** - If we add a folder via the GitLab UI, GitLab itself will add the `.gitkeep` to that new dir. + > **Note:** If we add a folder via the GitLab UI, GitLab itself will add the `.gitkeep` to that new dir. Now, let's run a local test and see if everything we did didn't break anything. @@ -248,64 +247,64 @@ project. - The fastest and easiest way to do this, is to click on **Set up CI** on project's main page: -  +  - On next screen, we can select a template ready to go. Click on **Apply a GitLab CI/CD Yaml template** and select **Elixir**: -  +  - This template file tells GitLab CI/CD about what we wish to do every time a new commit is made. - However, we have to adapt it to run a Phoenix app. + This template file tells GitLab CI/CD about what we wish to do every time a new commit is made. + However, we have to adapt it to run a Phoenix app. - The first line tells GitLab what Docker image will be used. - Remember when we learn about Runners, the isolated virtual machine where GitLab CI/CD build and test - our application? This virtual machine must have all dependencies to run our application. This is - where a Docker image is needed. The correct image will provide the entire system for us. + Remember when we learn about Runners, the isolated virtual machine where GitLab CI/CD build and test + our application? This virtual machine must have all dependencies to run our application. This is + where a Docker image is needed. The correct image will provide the entire system for us. - As a suggestion, you can use [trenpixster's elixir image][docker-image], which already has all - dependencies for Phoenix installed, such as Elixir, Erlang, NodeJS and PostgreSQL: + As a suggestion, you can use [trenpixster's elixir image][docker-image], which already has all + dependencies for Phoenix installed, such as Elixir, Erlang, NodeJS and PostgreSQL: - ```yml - image: trenpixster/elixir:latest - ``` + ```yml + image: trenpixster/elixir:latest + ``` - At `services` session, we'll only use `postgres`, so we'll delete `mysql` and `redis` lines: - ```yml - services: - - postgres:latest - ``` + ```yml + services: + - postgres:latest + ``` - Now, we'll create a new entry called `variables`, before `before_script` session: - ```yml - variables: - POSTGRES_DB: hello_gitlab_ci_test - POSTGRES_HOST: postgres - POSTGRES_USER: postgres - POSTGRES_PASSWORD: "postgres" - MIX_ENV: "test" - ``` + ```yml + variables: + POSTGRES_DB: hello_gitlab_ci_test + POSTGRES_HOST: postgres + POSTGRES_USER: postgres + POSTGRES_PASSWORD: "postgres" + MIX_ENV: "test" + ``` - Here, we are setting up the values for GitLab CI/CD authenticate into PostgreSQL, as we did on - `config/test.exs` earlier. + Here, we are setting up the values for GitLab CI/CD authenticate into PostgreSQL, as we did on + `config/test.exs` earlier. - In `before_script` session, we'll add some commands to prepare everything to the test: - ```yml - before_script: - - apt-get update && apt-get -y install postgresql-client - - mix local.hex --force - - mix deps.get --only test - - mix ecto.create - - mix ecto.migrate - ``` + ```yml + before_script: + - apt-get update && apt-get -y install postgresql-client + - mix local.hex --force + - mix deps.get --only test + - mix ecto.create + - mix ecto.migrate + ``` - It's important to install `postgresql-client` to let GitLab CI/CD access PostgreSQL and create our - database with the login information provided earlier. More important is to respect the indentation, - to avoid syntax errors when running the build. + It's important to install `postgresql-client` to let GitLab CI/CD access PostgreSQL and create our + database with the login information provided earlier. More important is to respect the indentation, + to avoid syntax errors when running the build. - And finally, we'll let `mix` session intact. @@ -404,14 +403,14 @@ other reasons][ci-reasons] to keep using GitLab CI/CD. The benefits to our teams [phoenix-learning-guide]: https://hexdocs.pm/phoenix/learning.html "Phoenix Learning Guide" [phoenix-install]: https://hexdocs.pm/phoenix/installation.html "Phoenix Installation" [phoenix-mysql]: https://hexdocs.pm/phoenix/ecto.html#using-mysql "Phoenix with MySQL" -[elixir-site]: http://elixir-lang.org/ "Elixir" -[elixir-mix]: http://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html "Introduction to mix" -[elixir-docs]: http://elixir-lang.org/getting-started/introduction.html "Elixir Documentation" +[elixir-site]: https://elixir-lang.org/ "Elixir" +[elixir-mix]: https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html "Introduction to mix" +[elixir-docs]: https://elixir-lang.org/getting-started/introduction.html "Elixir Documentation" [elixir-install]: https://elixir-lang.org/install.html "Elixir Installation" -[ecto]: http://hexdocs.pm/ecto "Ecto" +[ecto]: https://hexdocs.pm/ecto/Ecto.html "Ecto" [ecto-repo]: https://hexdocs.pm/ecto/Ecto.html#module-repositories "Ecto Repositories" [mix-ecto]: https://hexdocs.pm/ecto/Mix.Tasks.Ecto.Create.html "mix and Ecto" -[iex]: http://elixir-lang.org/getting-started/introduction.html#interactive-mode "Interactive Mode" +[iex]: https://elixir-lang.org/getting-started/introduction.html#interactive-mode "Interactive Mode" [ci-lint]: https://gitlab.com/ci/lint "CI Lint Tool" [ci-reasons]: https://about.gitlab.com/2015/02/03/7-reasons-why-you-should-be-using-ci/ "7 Reasons Why You Should Be Using CI" [ci-guide]: https://about.gitlab.com/2015/12/14/getting-started-with-gitlab-and-gitlab-ci/ "Getting Started With GitLab And GitLab CI/CD" diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 551044dd76f..cce33c7a6b4 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -69,24 +69,26 @@ correctly with your CI jobs: 1. Next, if you are using `gitlab-runner` v1.10+, you can set the `GIT_SUBMODULE_STRATEGY` variable to either `normal` or `recursive` to tell the runner to fetch your submodules before the job: - ```yaml - variables: - GIT_SUBMODULE_STRATEGY: recursive - ``` - See the [`.gitlab-ci.yml` reference](yaml/README.md#git-submodule-strategy) - for more details about `GIT_SUBMODULE_STRATEGY`. + + ```yaml + variables: + GIT_SUBMODULE_STRATEGY: recursive + ``` + + See the [`.gitlab-ci.yml` reference](yaml/README.md#git-submodule-strategy) + for more details about `GIT_SUBMODULE_STRATEGY`. 1. If you are using an older version of `gitlab-runner`, then use `git submodule sync/update` in `before_script`: - ```yaml - before_script: - - git submodule sync --recursive - - git submodule update --init --recursive - ``` + ```yaml + before_script: + - git submodule sync --recursive + - git submodule update --init --recursive + ``` - `--recursive` should be used in either both or none (`sync/update`) depending on - whether you have recursive submodules. + `--recursive` should be used in either both or none (`sync/update`) depending on + whether you have recursive submodules. The rationale to set the `sync` and `update` in `before_script` is because of the way Git submodules work. On a fresh Runner workspace, Git will set the diff --git a/doc/ci/img/collapsible_log.png b/doc/ci/img/collapsible_log.png Binary files differnew file mode 100644 index 00000000000..d2a570e246e --- /dev/null +++ b/doc/ci/img/collapsible_log.png diff --git a/doc/ci/img/deployments_view.png b/doc/ci/img/deployments_view.png Binary files differindex 12090434bef..9e2b7e89577 100644 --- a/doc/ci/img/deployments_view.png +++ b/doc/ci/img/deployments_view.png diff --git a/doc/ci/img/environments_available.png b/doc/ci/img/environments_available.png Binary files differindex 48fc6effc2d..6c64e9398f7 100644 --- a/doc/ci/img/environments_available.png +++ b/doc/ci/img/environments_available.png diff --git a/doc/ci/img/environments_mr_review_app.png b/doc/ci/img/environments_mr_review_app.png Binary files differindex 6a7b7ce5679..86c20d8d3b6 100644 --- a/doc/ci/img/environments_mr_review_app.png +++ b/doc/ci/img/environments_mr_review_app.png diff --git a/doc/ci/img/manual_job_variables.png b/doc/ci/img/manual_job_variables.png Binary files differnew file mode 100644 index 00000000000..a5ed351fdcd --- /dev/null +++ b/doc/ci/img/manual_job_variables.png diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 1387d4df500..58307660e51 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -8,7 +8,7 @@ type: reference Interactive web terminals give the user access to a terminal in GitLab for running one-off commands for their CI pipeline. Since this is giving the user -shell access to the environment where [GitLab Runner](https://docs.gitlab.com/runner/) +shell access to the environment where [GitLab Runner](https://docs.gitlab.com/runner/) is deployed, some [security precautions](../../administration/integration/terminal.md#security) were taken to protect the users. @@ -59,7 +59,6 @@ close the terminal window.  -## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]** +## Interactive Web Terminals for the Web IDE **(ULTIMATE ONLY)** Read the Web IDE docs to learn how to run [Interactive Terminals through the Web IDE](../../user/project/web_ide/index.md). - diff --git a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png Binary files differindex 204e9c462e5..f3fb9444b55 100644 --- a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png +++ b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png diff --git a/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png b/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png Binary files differindex 5089a1088c5..a0874f66eaa 100644 --- a/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png +++ b/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index ef9f9a9973c..366aca3442e 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -9,6 +9,11 @@ In this document we'll present an overview of the concepts of Continuous Integra Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD. +NOTE: **Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more.** +Watch our +["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) +webcast to learn about continuous methods and how GitLab’s built-in CI can help you simplify and scale software development. + ## Introduction to CI/CD methodologies The continuous methodologies of software development are based on @@ -146,14 +151,14 @@ so, GitLab CI/CD: - Runs automated scripts (sequential or parallel) to: - Build and test your app. - Preview the changes per merge request with Review Apps, as you - would see in your `localhost`. + would see in your `localhost`. Once you're happy with your implementation: - Get your code reviewed and approved. - Merge the feature branch into the default branch. - GitLab CI/CD deploys your changes automatically to a production environment. -- And finally, you and your team can easily roll it back if something goes wrong. +- And finally, you and your team can easily roll it back if something goes wrong.  @@ -176,24 +181,24 @@ you'll see some of the features available in GitLab according to each stage (Verify, Package, Release). 1. **Verify**: - - Automatically build and test your application with Continuous Integration. - - Analyze your source code quality with [GitLab Code Quality](../../user/project/merge_requests/code_quality.md). **[STARTER]** - - Determine the performance impact of code changes with [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md). **[PREMIUM]** - - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **[ULTIMATE]**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **[ULTIMATE]**, and [JUnit tests](../junit_test_reports.md). - - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. + - Automatically build and test your application with Continuous Integration. + - Analyze your source code quality with [GitLab Code Quality](../../user/project/merge_requests/code_quality.md). **(STARTER)** + - Determine the performance impact of code changes with [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md). **(PREMIUM)** + - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [JUnit tests](../junit_test_reports.md). + - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. 1. **Package**: - - Store Docker images with [Container Registry](../../user/project/container_registry.md). - - Store NPM packages with [NPM Registry](../../user/project/packages/npm_registry.md). **[PREMIUM]** - - Store Maven artifacts with [Maven Repository](../../user/project/packages/maven_repository.md). **[PREMIUM]** + - Store Docker images with [Container Registry](../../user/project/container_registry.md). + - Store NPM packages with [NPM Registry](../../user/project/packages/npm_registry.md). **(PREMIUM)** + - Store Maven artifacts with [Maven Repository](../../user/project/packages/maven_repository.md). **(PREMIUM)** 1. **Release**: - - Continuous Deployment, automatically deploying your app to production. - - Continuous Delivery, manually click to deploy your app to production. - - Deploy static websites with [GitLab Pages](../../user/project/pages/index.md). - - Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature with [Canary Deployments](../../user/project/canary_deployments.md). **[PREMIUM]** - - Deploy your features behind [Feature Flags](../../user/project/operations/feature_flags.md). **[PREMIUM]** - - Add release notes to any Git tag with [GitLab Releases](../../user/project/releases/index.md). - - View of the current health and status of each CI environment running on Kubernetes with [Deploy Boards](../../user/project/deploy_boards.md). **[PREMIUM]** - - Deploy your application to a production environment in a Kubernetes cluster with [Auto Deploy](../../topics/autodevops/index.md#auto-deploy). + - Continuous Deployment, automatically deploying your app to production. + - Continuous Delivery, manually click to deploy your app to production. + - Deploy static websites with [GitLab Pages](../../user/project/pages/index.md). + - Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature with [Canary Deployments](../../user/project/canary_deployments.md). **(PREMIUM)** + - Deploy your features behind [Feature Flags](../../user/project/operations/feature_flags.md). **(PREMIUM)** + - Add release notes to any Git tag with [GitLab Releases](../../user/project/releases/index.md). + - View of the current health and status of each CI environment running on Kubernetes with [Deploy Boards](../../user/project/deploy_boards.md). **(PREMIUM)** + - Deploy your application to a production environment in a Kubernetes cluster with [Auto Deploy](../../topics/autodevops/index.md#auto-deploy). With GitLab CI/CD you can also: @@ -201,7 +206,7 @@ With GitLab CI/CD you can also: - Deploy your app to different [environments](../environments.md). - Install your own [GitLab Runner](https://docs.gitlab.com/runner/). - [Schedule pipelines](../../user/project/pipelines/schedules.md). -- Check for app vulnerabilities with [Security Test reports](../../user/project/merge_requests/index.md#security-reports-ultimate). **[ULTIMATE]** +- Check for app vulnerabilities with [Security Test reports](../../user/project/merge_requests/index.md#security-reports-ultimate). **(ULTIMATE)** To see all CI/CD features, navigate back to the [CI/CD index](../README.md). diff --git a/doc/ci/jenkins/index.md b/doc/ci/jenkins/index.md new file mode 100644 index 00000000000..f8a3fab88e3 --- /dev/null +++ b/doc/ci/jenkins/index.md @@ -0,0 +1,232 @@ +--- +comments: false +type: index, howto +--- + +# Migrating from Jenkins + +A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. To make this +easier if you're just getting started, we've collected several resources here that you might find useful +before diving in. + +First of all, our [Quick Start Guide](../quick_start/README.md) contains a good overview of how GitLab CI/CD works. +You may also be interested in [Auto DevOps](../../topics/autodevops/index.md) which can potentially be used to build, test, +and deploy your applications with little to no configuration needed at all. + +Otherwise, read on for important information that will help you get the ball rolling. Welcome +to GitLab! + +## Important differences + +There are some high level differences between the products worth mentioning: + +- With GitLab you don't need a root `pipeline` keyword to wrap everything. +- All jobs within a single stage always run in parallel, and all stages run in sequence. We are planning + to allow certain jobs to break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-ce/issues/47063) + feature. +- The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but + is in the YAML format (see [complete reference](../yaml/README.md)) instead of a Groovy DSL. It's most + analagous to the declarative Jenkinsfile format. +- GitLab comes with a [container registry](../../user/project/container_registry.md), and we recommend using + container images to set up your build environment. + +## Groovy vs. YAML + +Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. +GitLab works a bit differently, we use the more highly structured [YAML](https://yaml.org/) format, which +places scripting elements inside of `script:` blocks separate from the pipeline specification itself. + +This is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running +and avoids some of the problem of unconstrained complexity which can make your Jenkinsfiles hard to understand +and manage. + +That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that +behaviors of your jobs can be codified once and applied as needed. You can use the `extends:` syntax to +[templatize your jobs](../yaml/README.md#extends), and `include:` can be used to [bring in entire sets of behaviors](../yaml/README.md#include) +to pipelines in different projects. + +```yaml +.in-docker: + tags: + - docker + image: alpine + +rspec: + extends: + - .in-docker + script: + - rake rspec +``` + +## Artifact publishing + +Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define +a set of artifacts to be saved by using the `artifacts:` keyword. This can be configured to point to a file +or set of files that can then be persisted from job to job. Read more on our detailed [artifacts documentation](../../user/project/pipelines/job_artifacts.html) + +```yaml +pdf: + script: xelatex mycv.tex + artifacts: + paths: + - ./mycv.pdf + - ./output/ + expire_in: 1 week +``` + +Additionally, we have package management features like a built-in container, NPM, and Maven registry that you +can leverage. You can see the complete list of packaging features (which includes links to documentation) +in the [Packaging section of our documentation](../../README.md#package). + +## Integrated features + +Where you may have used plugins to get things like code quality, unit tests, security scanning, and so on working in Jenkins, +GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into +your Merge Requests, pipeline details pages, and other locations. You may find that you actually don't +need to configure anything to have these appear. + +If they aren't working as expected, or if you'd like to see what's available, our [CI feature index](../README.md#feature-set) has the full list +of bundled features and links to the documentation for each. + +## Converting Declarative Jenkinsfiles + +Declarative Jenkinsfiles contain "Sections" and "Directives" which are used to control the behavior of your +pipelines. There are equivalents for all of these in GitLab, which we've documented below. + +This section is based on the [Jenkinsfile syntax documentation](https://jenkins.io/doc/book/pipeline/syntax/) +and is meant to be a mapping of concepts there to concepts in GitLab. + +### Sections + +#### `agent` + +The agent section is used to define how a pipeline will be executed. For GitLab, we use the [GitLab Runner](../runners/README.md) +to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage +of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users.) The link above will bring you to the documenation which will describe how to get +up and running quickly. We also support using [tags](../runners/README.md#using-tags) to direct different jobs +to different Runners (execution agents). + +The `agent` section also allows you to define which Docker images should be used for execution, for which we use +the [`image`](../yaml/README.md#image) keyword. The `image` can be set on a single job or at the top level, in which +case it will apply to all jobs in the pipeline. + +```yaml +my_job: + image: alpine + ... +``` + +#### `post` + +The `post` section defines the actions that should be performed at the end of the pipeline. GitLab also supports +this through the use of stages. You can define your stages as follows, and any jobs assigned to the `before_pipeline` +or `after_pipeline` stages will run as expected. You can call these stages anything you like. + +```yaml +stages: + - before_pipeline + - build + - test + - deploy + - after_pipeline +``` + +Setting a step to be performed before and after any job can be done via the [`before_script` and `after_script` keywords](../yaml/README.md#before_script-and-after_script). + +```yaml +default: + before_script: + - echo "I run before any jobs starts in the entire pipeline, and can be responsible for setting up the environment." +``` + +#### `stages` + +GitLab CI also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/README.md#stages) +is a top level setting that enumerates the list of stages, but you are not required to nest individual jobs underneath +the `stages` section. Any job defined in the `.gitlab-ci.yml` can be made a part of any stage through use of the +[`stage:` keyword](../yaml/README.md#stage). + +Note that, unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage +which are run in that order. Jobs that have no `stage` defined are placed by default in the `test` stage. +Of course, each job that refers to a stage must refer to a stage that exists in the pipeline configuration. + +```yaml +stages: + - build + - test + - deploy + +my_job: + stage: build + ... +``` + +#### `steps` + +The `steps` section is equivalent to the [`script` section](../yaml/README.md#script) of an individual job. This is +a simple YAML array with each line representing an individual command to be run. + +```yaml +my_job: + script: + - echo "hello! the current time is:" + - time + ... +``` + +### Directives + +#### `environment` + +In GitLab, we use the [`variables` keyword](../yaml/README.md#variables) to define different variables at runtime. +These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/README.md), +including the section on [protected variables](../variables/README.md#protected-environment-variables) which can be used +to limit access to certain variables to certain environments or runners. + +```yaml +variables: + POSTGRES_USER: user + POSTGRES_PASSWORD: testing_password +``` + +#### `options` + +Here, options for different things exist associated with the object in question itself. For example, options related +to jobs are defined in relation to the job itself. If you're looking for a certain option, you should be able to find +where it's located by searching our [complete configuration reference](../yaml/README.md) page. + +#### `parameters` + +GitLab does not require you to define which variables you want to be available when starting a manual job. A user +can provide any variables they like. + +#### `triggers` / `cron` + +Because GitLab is integrated tightly with git, SCM polling options for triggers are not needed. We support an easy to use +[syntax for scheduling pipelines](../../user/project/pipelines/schedules.md). + +#### `tools` + +GitLab does not support a separate `tools` directive. Our best-practice reccomendation is to use pre-built +container images, which can be cached, and can be built to already contain the tools you need for your pipelines. Pipelines can +be set up to automatically build these images as needed and deploy them to the [container registry](../../user/project/container_registry.md). + +If you're not using container images with Docker/Kubernetes, for example on Mac or FreeBSD, then the `shell` executor does require you to +set up your environment either in advance or as part of the jobs. You could create a `before_script` +action that handles this for you. + +#### `input` + +Similar to the `parameters` keyword, this is not needed because a manual job can always be provided runtime +variable entry. + +#### `when` + +GitLab does support a [`when` keyword](../yaml/README.md#when) which is used to indicate when a job should be +run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in +our very powerful [`only/except` rules system](../yaml/README.md#onlyexcept-basic) (see also our [advanced syntax](../yaml/README.md#onlyexcept-basic)) + +```yaml +my_job: + only: branches +```
\ No newline at end of file diff --git a/doc/ci/merge_request_pipelines/img/merge_request.png b/doc/ci/merge_request_pipelines/img/merge_request.png Binary files differindex d03fdc6a885..bb64e17cc91 100644 --- a/doc/ci/merge_request_pipelines/img/merge_request.png +++ b/doc/ci/merge_request_pipelines/img/merge_request.png diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png b/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png Binary files differdeleted file mode 100644 index 58d5581f628..00000000000 --- a/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png b/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png Binary files differdeleted file mode 100644 index 0a84e61d284..00000000000 --- a/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index fe2fc790505..72a9a876037 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -1,14 +1,9 @@ --- -type: reference +type: reference, index +last_update: 2019-07-03 --- -# Pipelines for merge requests - -NOTE: **Note**: -As of GitLab 11.10, pipelines for merge requests require GitLab Runner 11.9 -or higher due to the [recent refspecs -changes](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25504). -Anything lower will cause the pipeline to fail. +# Pipelines for Merge Requests > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/15310) in GitLab 11.6. @@ -23,6 +18,16 @@ for when you are running a pipeline in a merge request. This could be either adding or removing steps in the pipeline, to make sure that your pipelines are as efficient as possible. +## Requirements and limitations + +Pipelines for merge requests have the following requirements and limitations: + +- As of GitLab 11.10, pipelines for merge requests require GitLab Runner 11.9 + or higher due to the + [recent refspecs changes](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25504). +- Pipelines for merge requests are incompatible with + [CI/CD for external repositories](../ci_cd_for_external_repos/index.md). + ## Configuring pipelines for merge requests To configure pipelines for merge requests, add the `only: merge_requests` parameter to @@ -71,56 +76,13 @@ when a merge request was created or updated. For example:  -## Pipelines for Merged Results **[PREMIUM]** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. -> This feature is disabled by default until we resolve issues with [contention handling](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186), but [can be enabled manually](#enabling-pipelines-for-merged-results). - -It's possible for your source and target branches to diverge, which can result -in the scenario that source branch's pipeline was green, the target's pipeline was green, -but the combined output fails. - -By having your merge request pipeline automatically -create a new ref that contains the merge result of the source and target branch -(then running a pipeline on that ref), we can better test that the combined result -is also valid. - -GitLab can run pipelines for merge requests -on this merged result. That is, where the source and target branches are combined into a -new ref and a pipeline for this ref validates the result prior to merging. - - +## Pipelines for Merged Results **(PREMIUM)** -There are some cases where creating a combined ref is not possible or not wanted. -For example, a source branch that has conflicts with the target branch -or a merge request that is still in WIP status. In this case, the merge request pipeline falls back to a "detached" state -and runs on the source branch ref as if it was a regular pipeline. +Read the [documentation on Pipelines for Merged Results](pipelines_for_merged_results/index.md). -The detached state serves to warn you that you are working in a situation -subjected to merge problems, and helps to highlight that you should -get out of WIP status or resolve merge conflicts as soon as possible. +### Merge Trains **(PREMIUM)** -### Enabling Pipelines for Merged Results - -To enable pipelines on merged results at the project level: - -1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Merge pipelines will try to validate the post-merge result prior to merging**. -1. Click **Save changes** button. - - - -CAUTION: **Warning:** -Make sure your `gitlab-ci.yml` file is [configured properly for pipelines for merge requests](#configuring-pipelines-for-merge-requests), -otherwise pipelines for merged results won't run and your merge requests will be stuck in an unresolved state. - -### Pipelines for Merged Result's limitations - -- This feature requires [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. -- This feature requires [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. -- After the merge request pipeline succeeds, if the target branch has moved forward, the result of the pipeline is stale and must be retried. In busy repos, this can become a problem as it is highly probable that the target branch will have moved ahead. Improvements are [planned](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) for future versions of GitLab. -- Forking/cross-repo workflows are not currently supported. To follow progress, see [#9713](https://gitlab.com/gitlab-org/gitlab-ee/issues/9713). -- This feature is not available for [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). +Read the [documentation on Merge Trains](pipelines_for_merged_results/merge_trains/index.md). ## Excluding certain jobs @@ -206,3 +168,15 @@ to integrate your job with [GitLab Merge Request API](../../api/merge_requests.m You can find the list of available variables in [the reference sheet](../variables/predefined_variables.md). The variable names begin with the `CI_MERGE_REQUEST_` prefix. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png Binary files differnew file mode 100644 index 00000000000..6d4b66824e1 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png Binary files differnew file mode 100644 index 00000000000..3ee9d8ec93b --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md new file mode 100644 index 00000000000..ad07c662965 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -0,0 +1,93 @@ +--- +type: reference +last_update: 2019-07-03 +--- + +# Pipelines for Merged Results **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. + +It's possible for your source and target branches to diverge, which can result +in the scenario that source branch's pipeline was green, the target's pipeline was green, +but the combined output fails. + +By having your merge request pipeline automatically +create a new ref that contains the merge result of the source and target branch +(then running a pipeline on that ref), we can better test that the combined result +is also valid. + +GitLab can run pipelines for merge requests +on this merged result. That is, where the source and target branches are combined into a +new ref and a pipeline for this ref validates the result prior to merging. + + + +There are some cases where creating a combined ref is not possible or not wanted. +For example, a source branch that has conflicts with the target branch +or a merge request that is still in WIP status. In this case, +GitLab doesn't create a merge commit and the pipeline runs on source branch, instead, +which is a default behavior of [Pipelines for merge requests](../index.md) + i.e. `detached` label is shown to the pipelines. + +The detached state serves to warn you that you are working in a situation +subjected to merge problems, and helps to highlight that you should +get out of WIP status or resolve merge conflicts as soon as possible. + +## Requirements and limitations + +Pipelines for merged results require: + +- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. +- [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. + +In addition, pipelines for merged results have the following limitations: + +- Forking/cross-repo workflows are not currently supported. To follow progress, + see [#11934](https://gitlab.com/gitlab-org/gitlab-ee/issues/11934). +- This feature is not available for + [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. + To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). + +## Enabling Pipelines for Merged Results + +To enable pipelines on merged results at the project level: + +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. Check **Merge pipelines will try to validate the post-merge result prior to merging**. +1. Click **Save changes** button. + + + +CAUTION: **Warning:** +Make sure your `gitlab-ci.yml` file is [configured properly for pipelines for merge requests](../index.md#configuring-pipelines-for-merge-requests), +otherwise pipelines for merged results won't run and your merge requests will be stuck in an unresolved state. + +## Troubleshooting + +### Pipelines for merged results not created even with new change pushed to merge request + +Can be caused by some disabled feature flags. Please make sure that +the following feature flags are enabled on your GitLab instance: + +- `:ci_use_merge_request_ref` +- `:merge_ref_auto_sync` + +To check these feature flag values, please ask administrator to execute the following commands: + +```shell +> sudo gitlab-rails console # Login to Rails console of GitLab instance. +> Feature.enabled?(:ci_use_merge_request_ref) # Check if it's enabled or not. +> Feature.enable(:ci_use_merge_request_ref) # Enable the feature flag. +``` + +## Using Merge Trains **(PREMIUM)** + +By enabling [Pipelines for merged results](#pipelines-for-merged-results-premium), +GitLab will [automatically display](merge_trains/index.md#how-to-add-a-merge-request-to-a-merge-train) +a **Start/Add Merge Train button** as the most recommended merge strategy. + +Generally, this is a safer option than merging merge requests immediately as your +merge request will be evaluated with an expected post-merge result before the actual +merge happens. + +For more information, read the [documentation on Merge Trains](merge_trains/index.md). diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png Binary files differnew file mode 100644 index 00000000000..d7720ac1143 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png Binary files differnew file mode 100644 index 00000000000..9da959ad440 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_failure.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_failure.png Binary files differnew file mode 100644 index 00000000000..8a795fff432 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_failure.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png Binary files differnew file mode 100644 index 00000000000..03bc61129ba --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_position_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_position_v12_0.png Binary files differnew file mode 100644 index 00000000000..ec4b157d428 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_position_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png Binary files differnew file mode 100644 index 00000000000..a4d0c8cf0e6 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png Binary files differnew file mode 100644 index 00000000000..45762b8e85e --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md new file mode 100644 index 00000000000..80a1c264bc4 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -0,0 +1,144 @@ +--- +type: reference +last_update: 2019-07-03 +--- + +# Merge Trains **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. + +[Pipelines for merged results](../index.md#pipelines-for-merged-results-premium) introduces +running a build on the result of the merged code prior to merging, as a way to keep master green. + +There's a scenario, however, for teams with a high number of changes in the target branch (typically master) where in many or even all cases, +by the time the merged code is validated another commit has made it to master, invalidating the merged result. +You'd need some kind of queuing, cancellation or retry mechanism for these scenarios +in order to ensure an orderly flow of changes into the target branch. + +Each MR that joins a merge train joins as the last item in the train, +just as it works in the current state. However, instead of queuing and waiting, +each item takes the completed state of the previous (pending) merge ref, adds its own changes, +and starts the pipeline immediately in parallel under the assumption that everything is going to pass. + +In this way, if all the pipelines in the train merge successfully, no pipeline time is wasted either queuing or retrying. +If the button is subsequently pressed in a different MR, instead of creating a new pipeline for the target branch, +it creates a new pipeline targeting the merge result of the previous MR plus the target branch. +Pipelines invalidated through failures are immediately canceled and requeued. + +## Requirements and limitations + +Merge trains have the following requirements and limitations: + +- This feature requires that + [pipelines for merged results](../index.md#pipelines-for-merged-results-premium) are + **configured properly**. +- Each merge train can run a maximum of **four** pipelines in parallel. + If more than four merge requests are added to the merge train, the merge requests + will be queued until a slot in the merge train is free. There is no limit to the + number of merge requests that can be queued. +- This feature does not support [squash and merge](../../../../user/project/merge_requests/squash_and_merge.md). + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +Watch this video for a demonstration on [how parallel execution +of Merge Trains can prevent commits from breaking the default +branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). + +## How to add a merge request to a merge train + +To add a merge request to a merge train: + +1. Visit a merge request. +1. Click the **Start/Add to merge train** button. + + + +## How to remove a merge request from a merge train + +1. Visit a merge request. +1. Click the **Remove from merge train** button. + + + +## How to view a merge request's current position on the merge train + +After a merge request has been added to the merge train, the merge request's +current position will be displayed under the pipeline widget: + + + +## Start/Add to merge train when pipeline succeeds + +You can add a merge request to a merge train only when the latest pipeline in the +merge request finished. While the pipeline is running or pending, you cannot add +the merge request to a train because the current change of the merge request may +be broken thus it could affect the following merge requests. + +In this case, you can schedule to add the merge request to a merge train **when the latest +pipeline succeeds** (This pipeline is [Pipelines for merged results](../index.md), not Pipelines for merge train). +You can see the following button instead of the regular **Start/Add to merge train** +button while the latest pipeline is running. + + + +## Immediately merge a merge request with a merge train + +In case, you have a high-priority merge request (e.g. critical patch) to be merged urgently, +you can use **Merge Immediately** option for bypassing the merge train. +This is the fastest option to get the change merged into the target branch. + + + +However, every time you merge a merge request immediately, it could affect the +existing merge train to be reconstructed, specifically, it regenerates expected +merge commits and pipelines. This means, merging immediately essentially wastes +CI resources. + +## Troubleshooting + +### Merge request dropped from the merge train immediately + +If a merge request is not mergeable (for example, it's WIP, there is a merge +conflict, etc), your merge request will be dropped from the merge train automatically. + +In these cases, the reason for dropping the merge request is in the **system notes**. + +To check the reason: + +1. Open the merge request that was dropped from the merge train. +1. Open the **Discussion** tab. +1. Find a system note that includes either: + - The text **... removed this merge request from the merge train because ...** + - **... aborted this merge request from the merge train because ...** + The reason is given in the text after the **because ...** phrase. + + + +### Merge When Pipeline Succeeds cannot be chosen + +[Merge When Pipeline Succeeds](../../../../user/project/merge_requests/merge_when_pipeline_succeeds.md) +is unavailable when +[Pipelines for Merged Results is enabled](../index.md#enabling-pipelines-for-merged-results). + +Follow [this issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/12267) to +track progress on this issue. + +### Merge Train disturbs your workflow + +First of all, please check if [merge immediately](#immediately-merge-a-merge-request-with-a-merge-train) +is available as a workaround in your workflow. This is the most recommended +workaround you'd be able to take immediately. If it's not available or acceptable, +please read through this section. + +Merge train is enabled by default when you enable [Pipelines for merged results](../index.md), +however, you can forcibly disable this feature by disabling the feature flag `:merge_trains_enabled`. +After you disabled this feature, all the existing merge trains will be aborted and +you will no longer see the **Start/Add Merge Train** button in merge requests. + +To check if the feature flag is enabled on your GitLab instance, +please ask administrator to execute the following commands: + +```shell +> sudo gitlab-rails console # Login to Rails console of GitLab instance. +> Feature.enabled?(:merge_trains_enabled) # Check if it's enabled or not. +> Feature.disable(:merge_trains_enabled) # Disable the feature flag. +```
\ No newline at end of file diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index f9cfc0892a7..4d3f5a143f7 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -2,9 +2,9 @@ type: reference --- -# Metrics Reports **[PREMIUM]** +# Metrics Reports **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing) 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. Requires GitLab Runner 11.10 and above. ## Overview diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 50c8d82602b..cb8d383f7d9 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -2,7 +2,7 @@ type: reference --- -# Multi-project pipelines **[PREMIUM]** +# Multi-project pipelines **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/2121) in [GitLab Premium 9.3](https://about.gitlab.com/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs). @@ -171,6 +171,26 @@ In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the upstream pipeline will be passed to the `downstream-job` job, and will be available within the context of all downstream builds. +NOTE: **Tip:** +Upstream pipelines take precedence over downstream ones. If there are two +variables with the same name defined in both upstream and downstream projects, +the ones defined in the upstream project will take precedence. + +### Mirroring status from upstream pipeline + +You can mirror the pipeline status from an upstream pipeline to a bridge job by +using the `needs:pipeline` keyword. The latest pipeline status from master is +replicated to the bridge job. + +Example: + +```yaml +upstream_bridge: + stage: test + needs: + pipeline: other/project +``` + ### Limitations Because bridge jobs are a little different to regular jobs, it is not diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 1ad3d516df9..ed8d0e3bc35 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -6,6 +6,11 @@ type: reference > Introduced in GitLab 8.8. +NOTE: **Tip:** +Watch our +["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) +webcast to see a comprehensive demo of GitLab CI/CD pipeline. + ## Introduction Pipelines are the top-level component of continuous integration, delivery, and deployment. @@ -139,6 +144,20 @@ The union of A, B, and C is (1, 4) and (6, 7). Therefore, the total running time (4 - 1) + (7 - 6) => 4 ``` +### Expanding and collapsing job log sections + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/14664) in GitLab +> 12.0. + +Job logs are divided into sections that can be collapsed or expanded. + +In the following example: + +- Two sections are expanded and can be collapsed. +- One section is collapsed and can be expanded. + + + ## Configuring pipelines Pipelines, and their component jobs and stages, are defined in the [`.gitlab-ci.yml`](yaml/README.md) file for each project. @@ -220,7 +239,7 @@ Pipeline status and test coverage report badges are available and configurable f For information on adding pipeline badges to projects, see [Pipeline badges](../user/project/pipelines/settings.md#pipeline-badges). -## Multi-project pipelines **[PREMIUM]** +## Multi-project pipelines **(PREMIUM)** Pipelines for different projects can be combined and visualized together. @@ -304,6 +323,20 @@ stage has a job with a manual action.  +### Specifying variables when running manual jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30485) in GitLab 12.2. + +When running manual jobs you can supply additional job specific variables. + +You can do this from the job page of the manual job you want to run with +additional variables. + +This is useful when you want to alter the execution of a job by using +environment variables. + + + ### Delay a job in a pipeline graph > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767) in GitLab 11.4. @@ -331,7 +364,7 @@ GitLab provides API endpoints to: - [Triggering pipelines through the API](triggers/README.md). - [Pipeline triggers API](../api/pipeline_triggers.md). -### Start multiple manual actions in a stage +### Start multiple manual actions in a stage > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27188) in GitLab 11.11. diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 11bcfd5dc2c..5626428bfc3 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -13,6 +13,10 @@ NOTE: **Note:** Please keep in mind that only project Maintainers and Admin users have the permissions to access a project's settings. +NOTE: **Note:** +Coming over to GitLab from Jenkins? Check out our [reference](../jenkins/index.md) +for converting your pre-existing pipelines over to our format. + GitLab offers a [continuous integration][ci] service. If you [add a `.gitlab-ci.yml` file][yaml] to the root directory of your repository, and configure your GitLab project to use a [Runner], then each commit or @@ -233,7 +237,7 @@ CI with various languages. [runner-install]: https://docs.gitlab.com/runner/install/ [blog-ci]: https://about.gitlab.com/2015/05/06/why-were-replacing-gitlab-ci-jobs-with-gitlab-ci-dot-yml/ [examples]: ../examples/README.md -[ci]: https://about.gitlab.com/gitlab-ci/ +[ci]: https://about.gitlab.com/product/continuous-integration/ [yaml]: ../yaml/README.md [runner]: ../runners/README.md [enabled]: ../enable_or_disable_ci.md diff --git a/doc/ci/quick_start/img/build_log.png b/doc/ci/quick_start/img/build_log.png Binary files differindex 2bf0992c50e..16698629edc 100644 --- a/doc/ci/quick_start/img/build_log.png +++ b/doc/ci/quick_start/img/build_log.png diff --git a/doc/ci/quick_start/img/builds_status.png b/doc/ci/quick_start/img/builds_status.png Binary files differindex 58978e23978..b4aeeb988d2 100644 --- a/doc/ci/quick_start/img/builds_status.png +++ b/doc/ci/quick_start/img/builds_status.png diff --git a/doc/ci/quick_start/img/pipelines_status.png b/doc/ci/quick_start/img/pipelines_status.png Binary files differindex 06d1559f5d2..39a77a26b25 100644 --- a/doc/ci/quick_start/img/pipelines_status.png +++ b/doc/ci/quick_start/img/pipelines_status.png diff --git a/doc/ci/quick_start/img/runners_activated.png b/doc/ci/quick_start/img/runners_activated.png Binary files differindex cd83c1a7e4c..ac09e1d0137 100644 --- a/doc/ci/quick_start/img/runners_activated.png +++ b/doc/ci/quick_start/img/runners_activated.png diff --git a/doc/ci/review_apps/img/review_button.png b/doc/ci/review_apps/img/review_button.png Binary files differnew file mode 100644 index 00000000000..4f1cf4d7cfd --- /dev/null +++ b/doc/ci/review_apps/img/review_button.png diff --git a/doc/ci/review_apps/img/toolbar_feeback_form.png b/doc/ci/review_apps/img/toolbar_feeback_form.png Binary files differnew file mode 100644 index 00000000000..fe1c7e6e611 --- /dev/null +++ b/doc/ci/review_apps/img/toolbar_feeback_form.png diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 7b039fe6654..8ab7982fd65 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -27,7 +27,7 @@ In the above example: - Once the review as passed, `topic branch` is merged into `master` where it's deploy to staging. - After been approved in staging, the changes that were merged into `master` are deployed in to production. -### How Review Apps work +## How Review Apps work A Review App is a mapping of a branch with an [environment](../environments.md). Access to the Review App is made available as a link on the [merge request](../../user/project/merge_requests.md) relevant to the branch. @@ -41,27 +41,34 @@ In this example, a branch was: - Successfully built. - Deployed under a dynamic environment that can be reached by clicking on the **View app** button. +After adding Review Apps to your workflow, you follow the branched Git flow. That is: + +1. Push a branch and let the Runner deploy the Review App based on the `script` definition of the dynamic environment job. +1. Wait for the Runner to build and deploy your web application. +1. Click on the link provided in the merge request related to the branch to see the changes live. + ## Configuring Review Apps Review Apps are built on [dynamic environments](../environments.md#configuring-dynamic-environments), which allow you to dynamically create a new environment for each branch. The process of configuring Review Apps is as follows: -1. Set up the infrastructure to host and deploy the Review Apps. +1. Set up the infrastructure to host and deploy the Review Apps (check the [examples](#review-apps-examples) below). 1. [Install](https://docs.gitlab.com/runner/install/) and [configure](https://docs.gitlab.com/runner/commands/) a Runner to do deployment. 1. Set up a job in `.gitlab-ci.yml` that uses the [predefined CI environment variable](../variables/README.md) `${CI_COMMIT_REF_NAME}` to create dynamic environments and restrict it to run only on branches. 1. Optionally, set a job that [manually stops](../environments.md#stopping-an-environment) the Review Apps. -### Examples +## Review Apps examples The following are example projects that demonstrate Review App configuration: - [NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx). - [OpenShift](https://gitlab.com/gitlab-examples/review-apps-openshift). +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> See also the video [Demo: Cloud Native Development with GitLab](https://www.youtube.com/watch?v=jfIyQEwrocw), which includes a Review Apps example. -### Route Maps +## Route Maps > Introduced in GitLab 8.17. In GitLab 11.5, the file links are available in the merge request widget. @@ -82,7 +89,7 @@ To set up a route map, add a a file inside the repository at `.gitlab/route-map. which contains a YAML array that maps `source` paths (in the repository) to `public` paths (on the website). -#### Route Maps example +### Route Maps example The following is an example of a route map for [Middleman](https://middlemanapp.com), a static site generator (SSG) used to build [GitLab's website](https://about.gitlab.com), @@ -140,19 +147,108 @@ Once you have the route mapping set up, it will take effect in the following loc - In the diff for a merge request, comparison, or commit. -  +  - In the blob file view. -  +  -## Working with Review Apps +## Visual Reviews **(STARTER)** -After adding Review Apps to your workflow, you follow the branched Git flow. That is: +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10761) in GitLab Starter 12.0. -1. Push a branch and let the Runner deploy the Review App based on the `script` definition of the dynamic environment job. -1. Wait for the Runner to build and deploy your web application. -1. Click on the link that provided in the merge request related to the branch to see the changes live. +With Visual Reviews, you can provide a feedback form to your Review Apps so +that reviewers can post comments directly from the app back to the merge request +that spawned the Review App. + +### Configuring Visual Reviews + +The feedback form is served through a script you add to pages in your Review App. +If you have [Developer permissions](../../user/permissions.md) to the project, +you can access it by clicking the **Review** button in the **Pipeline** section +of the merge request. + + + +The provided script should be added to the `<head>` of you application and +consists of some project and merge request specific values. Here's what it +looks like: + +```html +<script + data-project-id='11790219' + data-merge-request-id='1' + data-mr-url='https://gitlab.example.com' + data-project-path='sarah/review-app-tester' + id='review-app-toolbar-script' + src='https://gitlab.example.com/assets/webpack/visual_review_toolbar.js'> +</script> +``` + +Ideally, you should use [environment variables](../variables/predefined_variables.md) +to replace those values at runtime when each review app is created: + +- `data-project-id` is the project ID, which can be found by the `CI_PROJECT_ID` + variable. +- `data-merge-request-id` is the merge request ID, which can be found by the + `CI_MERGE_REQUEST_IID` variable. `CI_MERGE_REQUEST_IID` is available only if + [`only: [merge_requests]`](../merge_request_pipelines/index.md) + is used and the merge request is created. +- `data-mr-url` is the URL of the GitLab instance and will be the same for all + review apps. +- `data-project-path` is the project's path, which can be found by `CI_PROJECT_PATH`. +- `id` is always `review-app-toolbar-script`, you don't need to change that. +- `src` is the source of the review toolbar script, which resides in the + respective GitLab instance and will be the same for all review apps. + +For example, in a Ruby application, you would need to have this script: + +```html +<script + data-project-id="ENV['CI_PROJECT_ID']" + data-merge-request-id="ENV['CI_MERGE_REQUEST_IID']" + data-mr-url='https://gitlab.example.com' + data-project-path="ENV['CI_PROJECT_PATH']" + id='review-app-toolbar-script' + src='https://gitlab.example.com/assets/webpack/visual_review_toolbar.js'> +</script> +``` + +Then, when your app is deployed via GitLab CI/CD, those variables should get +replaced with their real values. + +NOTE: **Note:** +Future enhancements [are planned](https://gitlab.com/gitlab-org/gitlab-ee/issues/11322) +to make this process even easier. + +### Using Visual Reviews + +After Visual Reviews has been [enabled](#configuring-visual-reviews) for the +Review App, the Visual Reviews feedback form is overlaid on the app's pages at +the bottom-right corner. + + + +To use the feedback form: + +1. Create a [personal access token](../../user/profile/personal_access_tokens.md) + with the API scope selected. +1. Paste the token into the feedback box when prompted. If you select **Remember me**, + your browser stores the token so that future visits to Review Apps at the same URL + will not require you to re-enter the token. To clear the token, click **Log out**. +1. Make a comment on the visual review. You can make use of all the + [Markdown annotations](../../user/markdown.md) that are also available in + merge request comments. +1. Finally, click **Send feedback**. + +After you make and submit a comment in the visual review box, it will appear +automatically in the respective merge request. + +TIP: **Tip:** +Because tokens must be entered on a per-domain basis and they can only be accessed +once, different review apps will not remember your token. You can save the token +to your password manager specifically for the purpose of Visual Reviews. This way, +you will not need to create additional tokens for each merge request. ## Limitations diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index b089229ab58..269bd5c3428 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -62,7 +62,7 @@ You can only register a shared Runner if you are an admin of the GitLab instance 1. Grab the shared-Runner token on the `admin/runners` page -  +  1. [Register the Runner][register] @@ -88,7 +88,7 @@ visit the project you want to make the Runner work for in GitLab: ## Registering a group Runner -Creating a group Runner requires Maintainer permissions for the group. To create a +Creating a group Runner requires Owner permissions for the group. To create a group Runner visit the group you want to make the Runner work for in GitLab: 1. Go to **Settings > CI/CD** to obtain the token @@ -124,9 +124,9 @@ To lock/unlock a Runner: ## Assigning a Runner to another project -If you are Maintainer on a project where a specific Runner is assigned to, and the +If you are an Owner on a project where a specific Runner is assigned to, and the Runner is not [locked only to that project](#locking-a-specific-runner-from-being-enabled-for-other-projects), -you can enable the Runner also on any other project where you have Maintainer permissions. +you can enable the Runner also on any other project where you have Owner permissions. To enable/disable a Runner in your project: @@ -250,7 +250,7 @@ When you [register a Runner][register], its default behavior is to **only pick** [tagged jobs](../yaml/README.md#tags). NOTE: **Note:** -Maintainer [permissions](../../user/permissions.md) are required to change the +Owner [permissions](../../user/permissions.md) are required to change the Runner settings. To make a Runner pick untagged jobs: @@ -319,21 +319,21 @@ How this feature will work: 1. You set the _maximum job timeout_ for a Runner to 24 hours 1. You set the _CI/CD Timeout_ for a project to **2 hours** 1. You start a job -1. The job, if running longer, will be timeouted after **2 hours** +1. The job, if running longer, will be timed out after **2 hours** **Example 2 - Runner timeout not configured** 1. You remove the _maximum job timeout_ configuration from a Runner 1. You set the _CI/CD Timeout_ for a project to **2 hours** 1. You start a job -1. The job, if running longer, will be timeouted after **2 hours** +1. The job, if running longer, will be timed out after **2 hours** **Example 3 - Runner timeout smaller than project timeout** 1. You set the _maximum job timeout_ for a Runner to **30 minutes** 1. You set the _CI/CD Timeout_ for a project to 2 hours 1. You start a job -1. The job, if running longer, will be timeouted after **30 minutes** +1. The job, if running longer, will be timed out after **30 minutes** ### Be careful with sensitive information @@ -373,12 +373,12 @@ attacker. To reset the token: -1. Go to **Settings > CI/CD** for a specified Project -1. Expand the **General pipelines settings** section -1. Find the **Runner token** form field and click the **Reveal value** button -1. Delete the value and save the form +1. Go to **Settings > CI/CD** for a specified Project. +1. Expand the **General pipelines settings** section. +1. Find the **Runner token** form field and click the **Reveal value** button. +1. Delete the value and save the form. 1. After the page is refreshed, expand the **Runners settings** section - and check the registration token - it should be changed + and check the registration token - it should be changed. From now on the old token is not valid anymore and will not allow to register a new Runner to the project. If you are using any tools to provision and @@ -418,9 +418,9 @@ You can find the IP address of a Runner for a specific project by:  -[install]: http://docs.gitlab.com/runner/install/ +[install]: https://docs.gitlab.com/runner/install/ [fifo]: https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics) -[register]: http://docs.gitlab.com/runner/register/ +[register]: https://docs.gitlab.com/runner/register/ [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md [project defined timeout]: ../../user/project/pipelines/settings.html#timeout diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index 697452cee83..9ea113969c8 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -25,6 +25,12 @@ variables: MYSQL_ROOT_PASSWORD: "<your_mysql_password>" ``` +NOTE: **Note:** +The `MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` variables can't be set in the GitLab UI. +To set them, assign them to a variable [in the UI](../variables/README.md#via-the-ui), +and then assign that variable to the +`MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` variables in your `.gitlab-ci.yml`. + And then configure your application to use the database, for example: ```yaml diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 211eea26eb0..142f4f262e5 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -25,6 +25,13 @@ variables: POSTGRES_PASSWORD: "" ``` +NOTE: **Note:** +The `POSTGRES_DB`, `POSTGRES_USER`, and `POSTGRES_PASSWORD` variables can't be set in +the GitLab UI. To set them, assign them to a variable +[in the UI](../variables/README.md#via-the-ui), and then assign that +variable to the `POSTGRES_DB`, `POSTGRES_USER`, and `POSTGRES_PASSWORD` variables in +your `.gitlab-ci.yml`. + And then configure your application to use the database, for example: ```yaml @@ -70,7 +77,7 @@ template1=# CREATE USER runner WITH PASSWORD '$password' CREATEDB; ``` *__Note:__ Notice that we created the user with the privilege to be able to -create databases (`CREATEDB`). In the following steps we will create a database +create databases (`CREATEDB`). In the following steps we will create a database explicitly for that user but having that privilege can be useful if in your testing framework you have tools that drop and create databases.* @@ -114,5 +121,5 @@ available [shared runners](../runners/README.md). Want to hack on it? Simply fork it, commit and push your changes. Within a few moments the changes will be picked by a public runner and the job will begin. -[hub-pg]: https://hub.docker.com/r/_/postgres/ +[hub-pg]: https://hub.docker.com/_/postgres [postgres-example-repo]: https://gitlab.com/gitlab-examples/postgres diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index 69591ed605c..d9f022a7125 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -57,44 +57,44 @@ to access it. This is where an SSH key pair comes in handy. 1. Modify your `.gitlab-ci.yml` with a `before_script` action. In the following example, a Debian based image is assumed. Edit to your needs: - ```yaml - before_script: - ## - ## Install ssh-agent if not already installed, it is required by Docker. - ## (change apt-get to yum if you use an RPM-based image) - ## - - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' - - ## - ## Run ssh-agent (inside the build environment) - ## - - eval $(ssh-agent -s) - - ## - ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store - ## We're using tr to fix line endings which makes ed25519 keys work - ## without extra base64 encoding. - ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556 - ## - - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null - - ## - ## Create the SSH directory and give it the right permissions - ## - - mkdir -p ~/.ssh - - chmod 700 ~/.ssh - - ## - ## Optionally, if you will be using any Git commands, set the user name and - ## and email. - ## - #- git config --global user.email "user@example.com" - #- git config --global user.name "User name" - ``` - - NOTE: **Note:** - The [`before_script`](../yaml/README.md#before_script-and-after_script) can be set globally - or per-job. + ```yaml + before_script: + ## + ## Install ssh-agent if not already installed, it is required by Docker. + ## (change apt-get to yum if you use an RPM-based image) + ## + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + + ## + ## Run ssh-agent (inside the build environment) + ## + - eval $(ssh-agent -s) + + ## + ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store + ## We're using tr to fix line endings which makes ed25519 keys work + ## without extra base64 encoding. + ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556 + ## + - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null + + ## + ## Create the SSH directory and give it the right permissions + ## + - mkdir -p ~/.ssh + - chmod 700 ~/.ssh + + ## + ## Optionally, if you will be using any Git commands, set the user name and + ## and email. + ## + #- git config --global user.email "user@example.com" + #- git config --global user.name "User name" + ``` + + NOTE: **Note:** + The [`before_script`](../yaml/README.md#before_script-and-after_script) can be set globally + or per-job. 1. Make sure the private server's [SSH host keys are verified](#verifying-the-ssh-host-keys). @@ -118,9 +118,9 @@ on, and use that key for all projects that are run on this machine. 1. Then from the terminal login as the `gitlab-runner` user: - ``` - sudo su - gitlab-runner - ``` + ``` + sudo su - gitlab-runner + ``` 1. Generate the SSH key pair as described in the instructions to [generate an SSH key](../../ssh/README.md#generating-a-new-ssh-key-pair). diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 04c541fefe7..2a382f18038 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -32,7 +32,7 @@ to protect trigger tokens. You can use the `CI_JOB_TOKEN` [variable][predef] (used to authenticate with the [GitLab Container Registry][registry]) in the following cases. -#### When used with multi-project pipelines **[PREMIUM]** +#### When used with multi-project pipelines **(PREMIUM)** > **Note**: The use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced][ee-2017] @@ -56,7 +56,7 @@ Pipelines triggered that way also expose a special variable: Read more about the [pipelines trigger API][trigapi]. -#### When a pipeline depends on the artifacts of another pipeline **[PREMIUM]** +#### When a pipeline depends on the artifacts of another pipeline **(PREMIUM)** > The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] in [GitLab Premium][ee] 9.5. @@ -97,17 +97,6 @@ overview of the time the triggers were last used.  -## Taking ownership of a trigger - -> **Note**: -GitLab 9.0 introduced a trigger ownership to solve permission problems. - -Each created trigger when run will impersonate their associated user including -their access to projects and their project permissions. - -You can take ownership of existing triggers by clicking *Take ownership*. -From now on the trigger will be run as you. - ## Revoking a trigger You can revoke a trigger any time by going at your project's @@ -282,8 +271,7 @@ Old triggers, created before GitLab 9.0 will be marked as legacy. Triggers with the legacy label do not have an associated user and only have access to the current project. They are considered deprecated and will be -removed with one of the future versions of GitLab. You are advised to -[take ownership](#taking-ownership-of-a-trigger) of any legacy triggers. +removed with one of the future versions of GitLab. [ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017 [ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346 diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index df455857dee..5a15b907da0 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -27,8 +27,7 @@ CI/CD's pipelines. Using variables means no hardcoded values. ### Predefined environment variables -GitLab CI/CD has a default set of -[predefined variables](predefined_variables.md) +GitLab CI/CD has a [default set of predefined variables](predefined_variables.md) which can be used without any specification needed. You can call issues numbers, user names, branch names, pipeline and commit IDs, and much more. @@ -36,7 +35,7 @@ pipeline and commit IDs, and much more. Predefined environment variables are the ones that GitLab provides out of the box for the local environment of the Runner. -GitLab reads the .gitlab-ci.yml file, sends the information +GitLab reads the `.gitlab-ci.yml` file, sends the information to the Runner (which runs the script commands), under which the variables are exposed. @@ -44,6 +43,9 @@ For example, two jobs under the same pipeline can share the same `CI_PIPELINE_ID` variable, but each one has its own `CI_JOB_ID` variable. +NOTE: **Note:** +Find here the full [**predefined variables reference table**](predefined_variables.md). + ### Custom environment variables When your use case requires a specific variable, you can @@ -92,7 +94,10 @@ This means that the value of the variable will be hidden in job logs, though it must match certain requirements to do so: - The value must be in a single line. -- The value must only consist of characters from the Base64 alphabet, defined in [RFC4648](https://tools.ietf.org/html/rfc4648). +- The value must only consist of characters from the Base64 alphabet (RFC4648). + + [In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-ce/issues/63043) + and newer, `@` and `:` are also valid values. - The value must be at least 8 characters long. - The value must not use variables. @@ -271,6 +276,7 @@ export CI_RUNNER_ID="10" export CI_RUNNER_DESCRIPTION="my runner" export CI_RUNNER_TAGS="docker, linux" export CI_SERVER="yes" +export CI_SERVER_HOST="example.com" export CI_SERVER_NAME="GitLab" export CI_SERVER_REVISION="70606bf" export CI_SERVER_VERSION="8.9.0" @@ -337,7 +343,7 @@ Group-level variables can be added by: 1. Navigating to your group's **Settings > CI/CD** page. 1. Inputing variable types, keys, and values in the **Variables** section. -Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. + Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. Once you set them, they will be available for all subsequent pipelines. @@ -367,8 +373,11 @@ variables take precedence over those defined in `.gitlab-ci.yml`. ## Unsupported variables There are cases where some variables cannot be used in the context of a -`.gitlab-ci.yml` definition (for example under `script`). Read more -about which variables are [not supported](where_variables_can_be_used.md). +`.gitlab-ci.yml` definition (for example under `script`). Read more about which variables are [not supported](where_variables_can_be_used.md). + +## Where variables can be used + +Click [here](where_variables_can_be_used.md) for a section that describes where and how the different types of variables can be used. ## Advanced use @@ -387,7 +396,7 @@ Protected variables can be added by going to your project's Once you set them, they will be available for all subsequent pipelines. -### Limiting environment scopes of environment variables **[PREMIUM]** +### Limiting environment scopes of environment variables You can limit the environment scope of a variable by [defining which environments][envs] it can be available for. @@ -478,81 +487,86 @@ Below you can find supported syntax reference: 1. Equality matching using a string - > Example: `$VARIABLE == "some value"` + Examples: - > Example: `$VARIABLE != "some value"` _(added in 11.11)_ + - `$VARIABLE == "some value"` + - `$VARIABLE != "some value"` (introduced in GitLab 11.11) - You can use equality operator `==` or `!=` to compare a variable content to a - string. We support both, double quotes and single quotes to define a string - value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` - are supported. `"some value" == $VARIABLE` is correct too. + You can use equality operator `==` or `!=` to compare a variable content to a + string. We support both, double quotes and single quotes to define a string + value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` + are supported. `"some value" == $VARIABLE` is correct too. 1. Checking for an undefined value - > Example: `$VARIABLE == null` + Examples: - > Example: `$VARIABLE != null` _(added in 11.11)_ + - `$VARIABLE == null` + - `$VARIABLE != null` (introduced in GitLab 11.11) - It sometimes happens that you want to check whether a variable is defined - or not. To do that, you can compare a variable to `null` keyword, like - `$VARIABLE == null`. This expression is going to evaluate to truth if - variable is not defined when `==` is used, or to falsey if `!=` is used. + It sometimes happens that you want to check whether a variable is defined + or not. To do that, you can compare a variable to `null` keyword, like + `$VARIABLE == null`. This expression is going to evaluate to truth if + variable is not defined when `==` is used, or to falsey if `!=` is used. 1. Checking for an empty variable - > Example: `$VARIABLE == ""` + Examples: - > Example: `$VARIABLE != ""` _(added in 11.11)_ + - `$VARIABLE == ""` + - `$VARIABLE != ""` (introduced in GitLab 11.11) - If you want to check whether a variable is defined, but is empty, you can - simply compare it against an empty string, like `$VAR == ''` or non-empty - string `$VARIABLE != ""`. + If you want to check whether a variable is defined, but is empty, you can + simply compare it against an empty string, like `$VAR == ''` or non-empty + string `$VARIABLE != ""`. 1. Comparing two variables - > Example: `$VARIABLE_1 == $VARIABLE_2` + Examples: - > Example: `$VARIABLE_1 != $VARIABLE_2` _(added in 11.11)_ + - `$VARIABLE_1 == $VARIABLE_2` + - `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) - It is possible to compare two variables. This is going to compare values - of these variables. + It is possible to compare two variables. This is going to compare values + of these variables. 1. Variable presence check - > Example: `$STAGING` - - If you only want to create a job when there is some variable present, - which means that it is defined and non-empty, you can simply use - variable name as an expression, like `$STAGING`. If `$STAGING` variable - is defined, and is non empty, expression will evaluate to truth. - `$STAGING` value needs to a string, with length higher than zero. - Variable that contains only whitespace characters is not an empty variable. + Example: `$STAGING` -1. Pattern matching _(added in 11.0)_ + If you only want to create a job when there is some variable present, + which means that it is defined and non-empty, you can simply use + variable name as an expression, like `$STAGING`. If `$STAGING` variable + is defined, and is non empty, expression will evaluate to truth. + `$STAGING` value needs to a string, with length higher than zero. + Variable that contains only whitespace characters is not an empty variable. - > Example: `$VARIABLE =~ /^content.*/` +1. Pattern matching (introduced in GitLab 11.0) - > Example: `$VARIABLE_1 !~ /^content.*/` _(added in 11.11)_ + Examples: - It is possible perform pattern matching against a variable and regular - expression. Expression like this evaluates to truth if matches are found - when using `=~`. It evaluates to truth if matches are not found when `!~` is used. + - `$VARIABLE =~ /^content.*/` + - `$VARIABLE_1 !~ /^content.*/` (introduced in GitLab 11.11) - Pattern matching is case-sensitive by default. Use `i` flag modifier, like - `/pattern/i` to make a pattern case-insensitive. + It is possible perform pattern matching against a variable and regular + expression. Expression like this evaluates to truth if matches are found + when using `=~`. It evaluates to truth if matches are not found when `!~` is used. -1. Conjunction / Disjunction + Pattern matching is case-sensitive by default. Use `i` flag modifier, like + `/pattern/i` to make a pattern case-insensitive. - > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` +1. Conjunction / Disjunction ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27925) in GitLab 12.0) - > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` + Examples: - > Example: `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` + - `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` + - `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` + - `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` - It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise - supported syntax may be used in a conjunctive or disjunctive statement. - Precedence of operators follows standard Ruby 2.5 operation - [precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html). + It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise + supported syntax may be used in a conjunctive or disjunctive statement. + Precedence of operators follows standard Ruby 2.5 operation + [precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html). ## Debug tracing @@ -642,6 +656,8 @@ Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-mach ++ CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace ++ export CI_SERVER=yes ++ CI_SERVER=yes +++ export 'CI_SERVER_HOST=example.com' +++ CI_SERVER_HOST='example.com' ++ export 'CI_SERVER_NAME=GitLab CI' ++ CI_SERVER_NAME='GitLab CI' ++ export CI_SERVER_VERSION= @@ -676,6 +692,8 @@ Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-mach ++ CI_JOB_NAME=debug_trace ++ export CI_JOB_STAGE=test ++ CI_JOB_STAGE=test +++ export CI_SERVER_HOST=example.com +++ CI_SERVER_HOST=example.com ++ export CI_SERVER_NAME=GitLab ++ CI_SERVER_NAME=GitLab ++ export CI_SERVER_VERSION=8.14.3-ee diff --git a/doc/ci/variables/img/custom_variables_output.png b/doc/ci/variables/img/custom_variables_output.png Binary files differindex 29f5c63b3d9..797e9ec07b9 100644 --- a/doc/ci/variables/img/custom_variables_output.png +++ b/doc/ci/variables/img/custom_variables_output.png diff --git a/doc/ci/variables/img/new_custom_variables_example.png b/doc/ci/variables/img/new_custom_variables_example.png Binary files differindex efe104efe4c..bb60e6bab21 100644 --- a/doc/ci/variables/img/new_custom_variables_example.png +++ b/doc/ci/variables/img/new_custom_variables_example.png diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differindex 3c8c59720cf..c77c5cb7764 100644 --- a/doc/ci/variables/img/override_variable_manual_pipeline.png +++ b/doc/ci/variables/img/override_variable_manual_pipeline.png diff --git a/doc/ci/variables/img/variable_types_usage_example.png b/doc/ci/variables/img/variable_types_usage_example.png Binary files differindex 0e8bde891fe..c2ae32fd048 100644 --- a/doc/ci/variables/img/variable_types_usage_example.png +++ b/doc/ci/variables/img/variable_types_usage_example.png diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 4655eec51de..409f7d62538 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -30,7 +30,7 @@ future GitLab releases.** | `CI_BUILDS_DIR` | all | 11.10 | Top-level directory where builds are executed. | | `CI_CONCURRENT_ID` | all | 11.10 | Unique ID of build execution within a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | Unique ID of build execution within a single executor and project. | -| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a push request. | +| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a merge request. Only populated when there is a merge request associated with the pipeline. | | `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | | `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. | | `CI_COMMIT_REF_NAME` | 9.0 | all | The branch or tag name for which project is built | @@ -78,7 +78,7 @@ future GitLab releases.** | `CI_PIPELINE_ID` | 8.10 | all | The unique id of the current pipeline that GitLab CI uses internally | | `CI_PIPELINE_IID` | 11.0 | all | The unique id of the current pipeline scoped to project | | `CI_PIPELINE_SOURCE` | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | -| `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered] | +| `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered](../triggers/README.md) | | `CI_PIPELINE_URL` | 11.1 | 0.5 | Pipeline details URL | | `CI_PROJECT_DIR` | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | | `CI_PROJECT_ID` | all | all | The unique id of the current project that GitLab CI uses internally | @@ -101,6 +101,7 @@ future GitLab releases.** | `CI_RUNNER_TAGS` | 8.10 | 0.5 | The defined runner tags | | `CI_RUNNER_VERSION` | all | 10.6 | GitLab Runner version that is executing the current job | | `CI_SERVER` | all | all | Mark that job is executed in CI environment | +| `CI_SERVER_HOST` | 12.1 | all | Host component of the GitLab instance URL, without protocol and port (like gitlab.example.com) | | `CI_SERVER_NAME` | all | all | The name of CI server that is used to coordinate jobs | | `CI_SERVER_REVISION` | all | all | GitLab revision that is used to schedule jobs | | `CI_SERVER_VERSION` | all | all | GitLab version that is used to schedule jobs | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 7d3f39a8c19..b5296f26269 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -89,13 +89,14 @@ Supported: - In `script`, it will work in the following lines of `script`. - In `after_script`, it will work in following lines of `after_script`. -Please notice the specific case of `after_script` scripts, that can: +In the case of `after_script` scripts, they can: -- Only use variables defined before the script within the same `after_script` section. +- Only use variables defined before the script within the same `after_script` + section. - Not use variables defined in `before_script` and `script`. -Both restrictions are caused by the fact, that `after_script` script is executed in a -[separated shell context](https://docs.gitlab.com/ee/ci/yaml/README.html#before_script-and-after_script). +These restrictions are because `after_script` scripts are executed in a +[separated shell context](../yaml/README.md#before_script-and-after_script). ## Persisted variables diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index b3aa1bcfad2..76b73729d0b 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -119,6 +119,35 @@ The following table lists available parameters for jobs: NOTE: **Note:** Parameters `types` and `type` are [deprecated](#deprecated-parameters). +## Setting default parameters + +Some parameters can be set globally as the default for all jobs using the +`default:` keyword. Default parameters can then be overridden by job-specific +configuration. + +The following job parameters can be defined inside a `default:` block: + +- [`image`](#image) +- [`services`](#services) +- [`before_script`](#before_script-and-after_script) +- [`after_script`](#before_script-and-after_script) +- [`cache`](#cache) + +In the following example, the `ruby:2.5` image is set as the default for all +jobs except the `rspec 2.6` job, which uses the `ruby:2.6` image: + +```yaml +default: + image: ruby:2.5 + +rspec: + script: bundle exec rspec + +rspec 2.6: + image: ruby:2.6 + script: bundle exec rspec +``` + ## Parameter details The following are detailed explanations for parameters used to configure CI/CD pipelines. @@ -215,27 +244,33 @@ This can be an array or a multi-line string. `after_script` is used to define the command that will be run after all jobs, including failed ones. This has to be an array or a multi-line string. -Script specified in `before_script` is: +Scripts specified in `before_script` are: -- Concatenated with script specified in the main `script`. Job-level `before_script` definition - override global-level `before_script` definition when concatenated with `script` definition. -- Executed together with main `script` script as one script in a single shell context. +- Concatenated with scripts specified in the main `script`. Job-level + `before_script` definition override global-level `before_script` definition + when concatenated with `script` definition. +- Executed together with main `script` script as one script in a single shell + context. -Script specified in `after_script`: +Scripts specified in `after_script`: - Have a current working directory set back to the default. -- Is executed in a shell context separated from `before_script` and `script` scripts. -- Because of separated context, cannot see changes done by scripts defined in `before_script` or `script` scripts: - - in shell - for example, command aliases and variables exported in `script` script, - - outside of the working tree (depending on the Runner executor) - for example, software installed - by a `before_script` or `script` script. +- Are executed in a shell context separated from `before_script` and `script` + scripts. +- Because of separated context, cannot see changes done by scripts defined + in `before_script` or `script` scripts, either: + - In shell. For example, command aliases and variables exported in `script` + scripts. + - Outside of the working tree (depending on the Runner executor). For example, + software installed by a `before_script` or `script` scripts. It's possible to overwrite the globally defined `before_script` and `after_script` if you set it per-job: ```yaml -before_script: - - global before script +default: + before_script: + - global before script job: before_script: @@ -470,7 +505,7 @@ Feature.enable(:allow_unsafe_ruby_regexp) ### `only`/`except` (advanced) CAUTION: **Warning:** -This an _alpha_ feature, and it is subject to change at any time without +This is an _alpha_ feature, and it is subject to change at any time without prior notice! GitLab supports both simple and complex strategies, so it's possible to use an @@ -968,6 +1003,8 @@ review_app: stop_review_app: stage: deploy + variables: + GIT_STRATEGY: none script: make delete-app when: manual environment: @@ -982,6 +1019,10 @@ Once the `review_app` job is successfully finished, it will trigger the set it up to `manual` so it will need a [manual action](#whenmanual) via GitLab's web interface in order to run. +Also in the example, `GIT_STRATEGY` is set to `none` so that GitLab Runner won’t +try to check out the code after the branch is deleted when the `stop_review_app` +job is [automatically triggered](../environments.md#automatically-stopping-an-environment). + The `stop_review_app` job is **required** to have the following keywords defined: - `when` - [reference](#when) @@ -1046,7 +1087,7 @@ globally and all jobs will use that definition. #### `cache:paths` Use the `paths` directive to choose which files or directories will be cached. -Wildcards can be used as well. +Wildcards can be used that follow the [glob](https://en.wikipedia.org/wiki/Glob_(programming)) patterns and [filepath.Match](https://golang.org/pkg/path/filepath/#Match). Cache all files in `binaries` that end in `.apk` and the `.config` file: @@ -1208,8 +1249,10 @@ be available for download in the GitLab UI. #### `artifacts:paths` -You can only use paths that are within the project workspace. To pass artifacts -between different jobs, see [dependencies](#dependencies). +You can only use paths that are within the project workspace. +Wildcards can be used that follow the [glob](https://en.wikipedia.org/wiki/Glob_(programming)) patterns and [filepath.Match](https://golang.org/pkg/path/filepath/#Match). + +To pass artifacts between different jobs, see [dependencies](#dependencies). Send all files in `binaries` and `.config`: @@ -1464,7 +1507,7 @@ concatenated into a single file. Use a filename pattern (`junit: rspec-*.xml`), an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). -##### `artifacts:reports:codequality` **[STARTER]** +##### `artifacts:reports:codequality` **(STARTER)** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1474,7 +1517,7 @@ as artifacts. The collected Code Quality report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests. -##### `artifacts:reports:sast` **[ULTIMATE]** +##### `artifacts:reports:sast` **(ULTIMATE)** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1485,7 +1528,7 @@ The collected SAST report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests, pipeline view and provide data for security dashboards. -##### `artifacts:reports:dependency_scanning` **[ULTIMATE]** +##### `artifacts:reports:dependency_scanning` **(ULTIMATE)** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1496,7 +1539,7 @@ The collected Dependency Scanning report will be uploaded to GitLab as an artifa be automatically shown in merge requests, pipeline view and provide data for security dashboards. -##### `artifacts:reports:container_scanning` **[ULTIMATE]** +##### `artifacts:reports:container_scanning` **(ULTIMATE)** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1507,7 +1550,7 @@ The collected Container Scanning report will be uploaded to GitLab as an artifac be automatically shown in merge requests, pipeline view and provide data for security dashboards. -##### `artifacts:reports:dast` **[ULTIMATE]** +##### `artifacts:reports:dast` **(ULTIMATE)** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1518,7 +1561,7 @@ The collected DAST report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests, pipeline view and provide data for security dashboards. -##### `artifacts:reports:license_management` **[ULTIMATE]** +##### `artifacts:reports:license_management` **(ULTIMATE)** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1529,7 +1572,7 @@ The collected License Management report will be uploaded to GitLab as an artifac be automatically shown in merge requests, pipeline view and provide data for security dashboards. -##### `artifacts:reports:performance` **[PREMIUM]** +##### `artifacts:reports:performance` **(PREMIUM)** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1539,7 +1582,7 @@ as artifacts. The collected Performance report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests. -##### `artifacts:reports:metrics` **[PREMIUM]** +##### `artifacts:reports:metrics` **(PREMIUM)** > Introduced in GitLab 11.10. @@ -1622,6 +1665,84 @@ You can ask your administrator to [flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) and bring back the old behavior. +### `needs` + +> Introduced in GitLab 12.2. + +The `needs:` keyword enables executing jobs out-of-order, allowing you to implement +a [directed acyclic graph](../directed_acyclic_graph/index.md) in your `.gitlab-ci.yml`. + +This lets you run some jobs without waiting for other ones, disregarding stage ordering +so you can have multiple stages running concurrently. + +Let's consider the following example: + +```yaml +linux:build: + stage: build + +mac:build: + stage: build + +linux:rspec: + stage: test + needs: [linux:build] + +linux:rubocop: + stage: test + needs: [linux:build] + +mac:rspec: + stage: test + needs: [mac:build] + +mac:rubocop: + stage: test + needs: [mac:build] + +production: + stage: deploy +``` + +This example creates three paths of execution: + +- Linux path: the `linux:rspec` and `linux:rubocop` jobs will be run as soon + as the `linux:build` job finishes without waiting for `mac:build` to finish. + +- macOS path: the `mac:rspec` and `mac:rubocop` jobs will be run as soon + as the `mac:build` job finishes, without waiting for `linux:build` to finish. + +- The `production` job will be executed as soon as all previous jobs + finish; in this case: `linux:build`, `linux:rspec`, `linux:rubocop`, + `mac:build`, `mac:rspec`, `mac:rubocop`. + +#### Requirements and limitations + +1. If `needs:` is set to point to a job that is not instantiated + because of `only/except` rules or otherwise does not exist, the + job will fail. +1. Note that on day one of the launch, we are temporarily limiting the + maximum number of jobs that a single job can need in the `needs:` array. Track + our [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7541) + for details on the current limit. +1. If you use `dependencies:` with `needs:`, it's important that you + do not mark a job as having a dependency on something that won't + have been run at the time it needs it. It's better to use both + keywords in this case so that GitLab handles the ordering appropriately. +1. It is impossible for now to have `needs: []` (empty needs), + the job always needs to depend on something, unless this is the job + in the first stage (see [gitlab-ce#65504](https://gitlab.com/gitlab-org/gitlab-ce/issues/65504)). +1. If `needs:` refers to a job that is marked as `parallel:`. + the current job will depend on all parallel jobs created. +1. `needs:` is similar to `dependencies:` in that needs to use jobs from + prior stages, this means that it is impossible to create circular + dependencies or depend on jobs in the current stage (see [gitlab-ce#65505](https://gitlab.com/gitlab-org/gitlab-ce/issues/65505)). +1. Related to the above, stages must be explicitly defined for all jobs + that have the keyword `needs:` or are referred to by one. +1. For self-managed users, the feature must be turned on using the `ci_dag_support` + feature flag. The `ci_dag_limit_needs` option, if set, will limit the number of + jobs that a single job can need to `50`. If unset, the limit is `5`. + ### `coverage` > [Introduced][ce-7447] in GitLab 8.17. @@ -1728,15 +1849,42 @@ sequentially from `job_name 1/N` to `job_name N/N`. For every job, `CI_NODE_INDEX` and `CI_NODE_TOTAL` [environment variables](../variables/README.md#predefined-environment-variables) are set. -A simple example: +Marking a job to be run in parallel requires only a simple addition to your configuration file: + +```diff + test: + script: rspec ++ parallel: 5 +``` +TIP: **Tip:** +Parallelize tests suites across parallel jobs. +Different languages have different tools to facilitate this. + +A simple example using [Sempahore Test Boosters](https://github.com/renderedtext/test-boosters) and RSpec to run some Ruby tests: + +```ruby +# Gemfile +source 'https://rubygems.org' + +gem 'rspec' +gem 'semaphore_test_boosters' +``` ```yaml test: - script: rspec - parallel: 5 + parallel: 3 + script: + - bundle + - bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL ``` -### `trigger` **[PREMIUM]** +CAUTION: **Caution:** +Please be aware that semaphore_test_boosters reports usages statistics to the author. + +You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec +job split into three separate jobs. + +### `trigger` **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. @@ -2208,10 +2356,10 @@ spinach: script: rake spinach ``` -It's also possible to use multiple parents for `extends`. -The algorithm used for merge is "closest scope wins", so keys -from the last member will always shadow anything defined on other levels. -For example: +In GitLab 12.0 and later, it's also possible to use multiple parents for +`extends`. The algorithm used for merge is "closest scope wins", so +keys from the last member will always shadow anything defined on other +levels. For example: ```yaml .only-important: @@ -2398,20 +2546,20 @@ There are three possible values: `none`, `normal`, and `recursive`: - `normal` means that only the top-level submodules will be included. It is equivalent to: - ``` - git submodule sync - git submodule update --init - ``` + ``` + git submodule sync + git submodule update --init + ``` - `recursive` means that all submodules (including submodules of submodules) will be included. This feature needs Git v1.8.1 and later. When using a GitLab Runner with an executor not based on Docker, make sure the Git version meets that requirement. It is equivalent to: - ``` - git submodule sync --recursive - git submodule update --init --recursive - ``` + ``` + git submodule sync --recursive + git submodule update --init --recursive + ``` Note that for this feature to work correctly, the submodules must be configured (in `.gitmodules`) with either: @@ -2537,18 +2685,39 @@ You can set it globally or per-job in the [`variables`](#variables) section. The following parameters are deprecated. -### `types` +### Globally-defined `types` CAUTION: **Deprecated:** `types` is deprecated, and could be removed in a future release. Use [`stages`](#stages) instead. -### `type` +### Job-defined `type` CAUTION: **Deprecated:** `type` is deprecated, and could be removed in one of the future releases. Use [`stage`](#stage) instead. +### Globally-defined `image`, `services`, `cache`, `before_script`, `after_script` + +Defining `image`, `services`, `cache`, `before_script`, and +`after_script` globally is deprecated. Support could be removed +from a future release. + +Use [`default:`](#setting-default-parameters) instead. For example: + +```yaml +default: + image: ruby:2.5 + services: + - docker:dind + cache: + paths: [vendor/] + before_script: + - bundle install --path vendor/ + after_script: + - rm -rf tmp/ +``` + ## Custom build directories > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1267) in Gitlab Runner 11.10 @@ -2615,6 +2784,24 @@ test: - pwd ``` +### Nested paths + +The value of `GIT_CLONE_PATH` is expanded once and nesting variables +within it is not supported. + +For example, you define both the variables below in your +`.gitlab-ci.yml` file: + +```yml +variables: + GOPATH: $CI_BUILDS_DIR/go + GIT_CLONE_PATH: $GOPATH/src/namespace/project +``` + +The value of `GIT_CLONE_PATH` is expanded once into +`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure +because `$CI_BUILDS_DIR` is not expanded. + ## Special YAML features It's possible to use special YAML features like anchors (`&`), aliases (`*`) @@ -2798,7 +2985,8 @@ Alternatively, one can pass the `ci.skip` [Git push option][push-option] if using Git 2.10 or newer: ```sh -git push -o ci.skip +git push --push-option=ci.skip # using git 2.10+ +git push -o ci.skip # using git 2.18+ ``` <!-- ## Troubleshooting |
