diff options
Diffstat (limited to 'doc/development')
| -rw-r--r-- | doc/development/README.md | 1 | ||||
| -rw-r--r-- | doc/development/automatic_ce_ee_merge.md | 31 | ||||
| -rw-r--r-- | doc/development/fe_guide/axios.md | 2 | ||||
| -rw-r--r-- | doc/development/file_storage.md | 108 | ||||
| -rw-r--r-- | doc/development/query_count_limits.md | 65 | ||||
| -rw-r--r-- | doc/development/ux_guide/index.md | 2 |
6 files changed, 191 insertions, 18 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index 12cca9f84b7..45e9565f9a7 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -75,6 +75,7 @@ comments: false - [Ordering table columns](ordering_table_columns.md) - [Verifying database capabilities](verifying_database_capabilities.md) - [Database Debugging and Troubleshooting](database_debugging.md) +- [Query Count Limits](query_count_limits.md) ## Testing guides diff --git a/doc/development/automatic_ce_ee_merge.md b/doc/development/automatic_ce_ee_merge.md index 5a784b6de06..cf6314f9521 100644 --- a/doc/development/automatic_ce_ee_merge.md +++ b/doc/development/automatic_ce_ee_merge.md @@ -5,17 +5,32 @@ Enterprise Edition (look for the [`CE Upstream` merge requests]). This merge is done automatically in a [scheduled pipeline](https://gitlab.com/gitlab-org/release-tools/-/jobs/43201679). -If a merge is already in progress, the job [doesn't create a new one](https://gitlab.com/gitlab-org/release-tools/-/jobs/43157687). -**If you are pinged in a `CE Upstream` merge request to resolve a conflict, -please resolve the conflict as soon as possible or ask someone else to do it!** - ->**Note:** -It's ok to resolve more conflicts than the one that you are asked to resolve. In -that case, it's a good habit to ask for a double-check on your resolution by -someone who is familiar with the code you touched. +## What to do if you are pinged in a `CE Upstream` merge request to resolve a conflict? + +1. Please resolve the conflict as soon as possible or ask someone else to do it + - It's ok to resolve more conflicts than the one that you are asked to resolve. + In that case, it's a good habit to ask for a double-check on your resolution + by someone who is familiar with the code you touched. +1. Once you have resolved your conflicts, push to the branch (no force-push) +1. Assign the merge request to the next person that has to resolve a conflict +1. If all conflicts are resolved after your resolution is pushed, keep the merge + request assigned to you: **you are now responsible for the merge request to be + green** +1. If you need any help, you can ping the current [release managers], or ask in + the `#ce-to-ee` Slack channel + +A few notes about the automatic CE->EE merge job: + +- If a merge is already in progress, the job + [doesn't create a new one](https://gitlab.com/gitlab-org/release-tools/-/jobs/43157687). +- If there is nothing to merge (i.e. EE is up-to-date with CE), the job doesn't + create a new one +- The job posts messages to the `#ce-to-ee` Slack channel to inform what's the + current CE->EE merge status (e.g. "A new MR has been created", "A MR is still pending") [`CE Upstream` merge requests]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests?label_name%5B%5D=CE+upstream +[release managers]: https://about.gitlab.com/release-managers/ ## Always merge EE merge requests before their CE counterparts diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index dcd8f5cb839..0d9397c3bd5 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -63,7 +63,7 @@ We have also decided against using [axios interceptors] because they are not sui }); afterEach(() => { - mock.reset(); + mock.restore(); }); ``` diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index cf00e24e11a..34a02bd2c3c 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -14,9 +14,9 @@ There are many places where file uploading is used, according to contexts: - User snippet attachments * Project - Project avatars - - Issues/MR Markdown attachments - - Issues/MR Legacy Markdown attachments - - CI Build Artifacts + - Issues/MR/Notes Markdown attachments + - Issues/MR/Notes Legacy Markdown attachments + - CI Artifacts (archive, metadata, trace) - LFS Objects @@ -25,7 +25,7 @@ There are many places where file uploading is used, according to contexts: GitLab started saving everything on local disk. While directory location changed from previous versions, they are still not 100% standardized. You can see them below: -| Description | In DB? | Relative path | Uploader class | model_type | +| Description | In DB? | Relative path (from CarrierWave.root) | Uploader class | model_type | | ------------------------------------- | ------ | ----------------------------------------------------------- | ---------------------- | ---------- | | Instance logo | yes | uploads/-/system/appearance/logo/:id/:filename | `AttachmentUploader` | Appearance | | Header logo | yes | uploads/-/system/appearance/header_logo/:id/:filename | `AttachmentUploader` | Appearance | @@ -33,17 +33,107 @@ they are still not 100% standardized. You can see them below: | User avatars | yes | uploads/-/system/user/avatar/:id/:filename | `AvatarUploader` | User | | User snippet attachments | yes | uploads/-/system/personal_snippet/:id/:random_hex/:filename | `PersonalFileUploader` | Snippet | | Project avatars | yes | uploads/-/system/project/avatar/:id/:filename | `AvatarUploader` | Project | -| Issues/MR Markdown attachments | yes | uploads/:project_path_with_namespace/:random_hex/:filename | `FileUploader` | Project | -| Issues/MR Legacy Markdown attachments | no | uploads/-/system/note/attachment/:id/:filename | `AttachmentUploader` | Note | -| CI Artifacts (CE) | yes | shared/artifacts/:year_:month/:project_id/:id | `ArtifactUploader` | Ci::Build | +| Issues/MR/Notes Markdown attachments | yes | uploads/:project_path_with_namespace/:random_hex/:filename | `FileUploader` | Project | +| Issues/MR/Notes Legacy Markdown attachments | no | uploads/-/system/note/attachment/:id/:filename | `AttachmentUploader` | Note | +| CI Artifacts (CE) | yes | shared/artifacts/:disk_hash[0..1]/:disk_hash[2..3]/:disk_hash/:year_:month_:date/:job_id/:job_artifact_id (:disk_hash is SHA256 digest of project_id) | `JobArtifactUploader` | Ci::JobArtifact | | LFS Objects (CE) | yes | shared/lfs-objects/:hex/:hex/:object_hash | `LfsObjectUploader` | LfsObject | CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader` -while in EE they inherit the `ObjectStoreUploader` and store files in and S3 API compatible object store. +while in EE they inherit the `ObjectStorage` and store files in and S3 API compatible object store. -In the case of Issues/MR Markdown attachments, there is a different approach using the [Hashed Storage] layout, +In the case of Issues/MR/Notes Markdown attachments, there is a different approach using the [Hashed Storage] layout, instead of basing the path into a mutable variable `:project_path_with_namespace`, it's possible to use the hash of the project ID instead, if project migrates to the new approach (introduced in 10.2). +### Path segments + +Files are stored at multiple locations and use different path schemes. +All the `GitlabUploader` derived classes should comply with this path segment schema: + +``` +| GitlabUploader +| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- | +| `<gitlab_root>/public/` | `uploads/-/system/` | `user/avatar/:id/` | `:filename` | +| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- | +| `CarrierWave.root` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` | +| | `CarrierWave::Uploader#store_dir` | | + +| FileUploader +| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- | +| `<gitlab_root>/shared/` | `artifacts/` | `:year_:month/:id` | `:filename` | +| `<gitlab_root>/shared/` | `snippets/` | `:secret/` | `:filename` | +| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- | +| `CarrierWave.root` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` | +| | `CarrierWave::Uploader#store_dir` | | +| | | `FileUploader#upload_path | + +| ObjectStore::Concern (store = remote) +| ----------------------- + ------------------------- + ----------------------------------- + -------------------------------- | +| `<bucket_name>` | <ignored> | `user/avatar/:id/` | `:filename` | +| ----------------------- + ------------------------- + ----------------------------------- + -------------------------------- | +| `#fog_dir` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` | +| | | `ObjectStorage::Concern#store_dir` | | +| | | `ObjectStorage::Concern#upload_path | +``` + +The `RecordsUploads::Concern` concern will create an `Upload` entry for every file stored by a `GitlabUploader` persisting the dynamic parts of the path using +`GitlabUploader#dynamic_path`. You may then use the `Upload#build_uploader` method to manipulate the file. + +## Object Storage + +By including the `ObjectStorage::Concern` in the `GitlabUploader` derived class, you may enable the object storage for this uploader. To enable the object storage +in your uploader, you need to either 1) include `RecordsUpload::Concern` and prepend `ObjectStorage::Extension::RecordsUploads` or 2) mount the uploader and create a new field named `<mount>_store`. + +The `CarrierWave::Uploader#store_dir` is overriden to + + - `GitlabUploader.base_dir` + `GitlabUploader.dynamic_segment` when the store is LOCAL + - `GitlabUploader.dynamic_segment` when the store is REMOTE (the bucket name is used to namespace) + +### Using `ObjectStorage::Extension::RecordsUploads` + +> Note: this concern will automatically include `RecordsUploads::Concern` if not already included. + +The `ObjectStorage::Concern` uploader will search for the matching `Upload` to select the correct object store. The `Upload` is mapped using `#store_dirs + identifier` for each store (LOCAL/REMOTE). + +```ruby +class SongUploader < GitlabUploader + include RecordsUploads::Concern + include ObjectStorage::Concern + prepend ObjectStorage::Extension::RecordsUploads + + ... +end + +class Thing < ActiveRecord::Base + mount :theme, SongUploader # we have a great theme song! + + ... +end +``` + +### Using a mounted uploader + +The `ObjectStorage::Concern` will query the `model.<mount>_store` attribute to select the correct object store. +This column must be present in the model schema. + +```ruby +class SongUploader < GitlabUploader + include ObjectStorage::Concern + + ... +end + +class Thing < ActiveRecord::Base + attr_reader :theme_store # this is an ActiveRecord attribute + mount :theme, SongUploader # we have a great theme song! + + def theme_store + super || ObjectStorage::Store::LOCAL + end + + ... +end +``` + [CarrierWave]: https://github.com/carrierwaveuploader/carrierwave [Hashed Storage]: ../administration/repository_storage_types.md diff --git a/doc/development/query_count_limits.md b/doc/development/query_count_limits.md new file mode 100644 index 00000000000..ebb6e0c2dac --- /dev/null +++ b/doc/development/query_count_limits.md @@ -0,0 +1,65 @@ +# Query Count Limits + +Each controller or API endpoint is allowed to execute up to 100 SQL queries. In +a production environment we'll only log an error in case this threshold is +exceeded, but in a test environment we'll raise an error instead. + +## Solving Failing Tests + +When a test fails because it executes more than 100 SQL queries there are two +solutions to this problem: + +1. Reduce the number of SQL queries that are executed. +2. Whitelist the controller or API endpoint. + +You should only resort to whitelisting when an existing controller or endpoint +is to blame as in this case reducing the number of SQL queries can take a lot of +effort. Newly added controllers and endpoints are not allowed to execute more +than 100 SQL queries and no exceptions will be made for this rule. _If_ a large +number of SQL queries is necessary to perform certain work it's best to have +this work performed by Sidekiq instead of doing this directly in a web request. + +## Whitelisting + +In the event that you _have_ to whitelist a controller you'll first need to +create an issue. This issue should (preferably in the title) mention the +controller or endpoint and include the appropriate labels (`database`, +`performance`, and at least a team specific label such as `Discussion`). + +Once the issue has been created you can whitelist the code in question. For +Rails controllers it's best to create a `before_action` hook that runs as early +as possible. The called method in turn should call +`Gitlab::QueryLimiting.whitelist('issue URL here')`. For example: + +```ruby +class MyController < ApplicationController + before_action :whitelist_query_limiting, only: [:show] + + def index + # ... + end + + def show + # ... + end + + def whitelist_query_limiting + Gitlab::QueryLimiting.whitelist('https://gitlab.com/gitlab-org/...') + end +end +``` + +By using a `before_action` you don't have to modify the controller method in +question, reducing the likelihood of merge conflicts. + +For Grape API endpoints there unfortunately is not a reliable way of running a +hook before a specific endpoint. This means that you have to add the whitelist +call directly into the endpoint like so: + +```ruby +get '/projects/:id/foo' do + Gitlab::QueryLimiting.whitelist('...') + + # ... +end +``` diff --git a/doc/development/ux_guide/index.md b/doc/development/ux_guide/index.md index 42bcf234e12..c59e7b72a1a 100644 --- a/doc/development/ux_guide/index.md +++ b/doc/development/ux_guide/index.md @@ -1,3 +1,5 @@ +> We are in the process of transferring UX documentation to the [design.gitlab.com](https://gitlab.com/gitlab-org/design.gitlab.com) project. Any updates to these docs should be made in that project. If documentation does not yet exist within [design.gitlab.com](https://gitlab.com/gitlab-org/design.gitlab.com), [create an issue](https://gitlab.com/gitlab-org/design.gitlab.com/issues) and merge request to add your new changes. + # GitLab UX Guide The goal of this guide is to provide standards, principles and in-depth information to design beautiful and effective GitLab features. This will be a living document, and we welcome contributions, feedback and suggestions. |