<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/gitlab/gitlab-ce.git/app/models/cycle_analytics, branch rc/enable-PreferredHashMethods-cop</title>
<subtitle>gitlab.com: gitlab-org/gitlab-ce.git
</subtitle>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/'/>
<entry>
<title>added new summary serializers and refactor all of the summary stuff into separate logical classes</title>
<updated>2017-01-17T10:32:54+00:00</updated>
<author>
<name>James Lopez</name>
<email>james@jameslopez.es</email>
</author>
<published>2016-11-22T10:46:02+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=fc6f8f20562ad761c034ffff076d329a3e9e8f4d'/>
<id>fc6f8f20562ad761c034ffff076d329a3e9e8f4d</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Use configured git rather than system git</title>
<updated>2017-01-05T12:46:26+00:00</updated>
<author>
<name>Robert Schilling</name>
<email>rschilling@student.tugraz.at</email>
</author>
<published>2017-01-05T12:46:26+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=22a34c81aa3801e5b453fd2c0a9364917e0aedbe'/>
<id>22a34c81aa3801e5b453fd2c0a9364917e0aedbe</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Merge branch 'jej-fix-missing-access-check-on-issues' into 'security'</title>
<updated>2016-11-29T00:25:46+00:00</updated>
<author>
<name>Douwe Maan</name>
<email>douwe@gitlab.com</email>
</author>
<published>2016-11-18T13:51:52+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=6d37fe952b5679d7586eaa569d0488dbb92032fe'/>
<id>6d37fe952b5679d7586eaa569d0488dbb92032fe</id>
<content type='text'>
Fix missing access checks on issue lookup using IssuableFinder

Split from !2024 to partially solve https://gitlab.com/gitlab-org/gitlab-ce/issues/23867

:warning: - Potentially untested
:bomb: - No test coverage
:traffic_light: - Test coverage of some sort exists (a test failed when error raised)
:vertical_traffic_light: - Test coverage of return value (a test failed when nil used)
:white_check_mark: - Permissions check tested

- [x] :white_check_mark: app/controllers/projects/branches_controller.rb:39
  - `before_action :authorize_push_code!` helpes limit/prevent exploitation. Always checks for reporter access so fine with
    confidential issues, issues only visible to team, etc.
- [x] :traffic_light: app/models/cycle_analytics/summary.rb:9 [`.count`]
- [x] :white_check_mark: app/controllers/projects/todos_controller.rb:19

- [x] Potential double render in app/controllers/projects/todos_controller.rb

- https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#cedccb227af9bfdf88802767cb58d43c2b977439_24_24

See merge request !2030
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Fix missing access checks on issue lookup using IssuableFinder

Split from !2024 to partially solve https://gitlab.com/gitlab-org/gitlab-ce/issues/23867

:warning: - Potentially untested
:bomb: - No test coverage
:traffic_light: - Test coverage of some sort exists (a test failed when error raised)
:vertical_traffic_light: - Test coverage of return value (a test failed when nil used)
:white_check_mark: - Permissions check tested

- [x] :white_check_mark: app/controllers/projects/branches_controller.rb:39
  - `before_action :authorize_push_code!` helpes limit/prevent exploitation. Always checks for reporter access so fine with
    confidential issues, issues only visible to team, etc.
- [x] :traffic_light: app/models/cycle_analytics/summary.rb:9 [`.count`]
- [x] :white_check_mark: app/controllers/projects/todos_controller.rb:19

- [x] Potential double render in app/controllers/projects/todos_controller.rb

- https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#cedccb227af9bfdf88802767cb58d43c2b977439_24_24

See merge request !2030
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix the "Commits" section of the cycle analytics summary.</title>
<updated>2016-09-26T13:08:40+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-26T04:33:57+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=c0ecef79fe936b9e71384f8d951fc5503f26f4a7'/>
<id>c0ecef79fe936b9e71384f8d951fc5503f26f4a7</id>
<content type='text'>
- The commit count was capped at 10, due to
  `Gitlab::Git::Repository#log` enforcing a limit, with the default set
  to 10.

- Reimplement a small portion of this `log` function to get just the
  data we need.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- The commit count was capped at 10, due to
  `Gitlab::Git::Repository#log` enforcing a limit, with the default set
  to 10.

- Reimplement a small portion of this `log` function to get just the
  data we need.
</pre>
</div>
</content>
</entry>
<entry>
<title>Implement a second round of review comments from @DouweM.</title>
<updated>2016-09-20T19:17:37+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-20T19:17:37+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=918e589c2b29c18d9fe3a8e6c93a3f490c86beb1'/>
<id>918e589c2b29c18d9fe3a8e6c93a3f490c86beb1</id>
<content type='text'>
- Don't use `TableReferences` - using `.arel_table` is shorter!
- Move some database-related code to `Gitlab::Database`
- Remove the `MergeRequest#issues_closed` and
  `Issue#closed_by_merge_requests`  associations. They were either
  shadowing or were too similar to existing methods. They are not being
  used anywhere, so it's better to remove them to reduce confusion.
- Use Rails 3-style validations
- Index for `MergeRequest::Metrics#first_deployed_to_production_at`
- Only include `CycleAnalyticsHelpers::TestGeneration` for specs that
  need it.
- Other minor refactorings.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- Don't use `TableReferences` - using `.arel_table` is shorter!
- Move some database-related code to `Gitlab::Database`
- Remove the `MergeRequest#issues_closed` and
  `Issue#closed_by_merge_requests`  associations. They were either
  shadowing or were too similar to existing methods. They are not being
  used anywhere, so it's better to remove them to reduce confusion.
- Use Rails 3-style validations
- Index for `MergeRequest::Metrics#first_deployed_to_production_at`
- Only include `CycleAnalyticsHelpers::TestGeneration` for specs that
  need it.
- Other minor refactorings.
</pre>
</div>
</content>
</entry>
<entry>
<title>Implement review comments from @yorickpeterse</title>
<updated>2016-09-20T10:35:25+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-20T09:36:54+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=8957293d9bd0d711db3af26182205c2fe4125194'/>
<id>8957293d9bd0d711db3af26182205c2fe4125194</id>
<content type='text'>
1. Change multiple updates to a single `update_all`

2. Use cascading deletes

3. Extract an average function for the database median.

4. Move database median to `lib/gitlab/database`

5. Use `delete_all` instead of `destroy_all`

6. Minor refactoring
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Change multiple updates to a single `update_all`

2. Use cascading deletes

3. Extract an average function for the database median.

4. Move database median to `lib/gitlab/database`

5. Use `delete_all` instead of `destroy_all`

6. Minor refactoring
</pre>
</div>
</content>
</entry>
<entry>
<title>Test the `CycleAnalytics::Summary` model.</title>
<updated>2016-09-20T07:53:14+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-20T04:24:42+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=204419e9e9b01977cbeac34b1a8385239b1455aa'/>
<id>204419e9e9b01977cbeac34b1a8385239b1455aa</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Make changes to the cycle analytics JSON endpoint.</title>
<updated>2016-09-19T09:30:55+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-19T09:30:55+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=4531e433ad33e67cb66abaa4626b5e608eb11f11'/>
<id>4531e433ad33e67cb66abaa4626b5e608eb11f11</id>
<content type='text'>
1. Add `summary` section.
2. `stats` is `null` if no stats are present.
3. `stats` and `summary` are both arrays.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Add `summary` section.
2. `stats` is `null` if no stats are present.
3. `stats` and `summary` are both arrays.
</pre>
</div>
</content>
</entry>
<entry>
<title>Move cycle analytics calculations to SQL.</title>
<updated>2016-09-17T06:46:48+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-16T06:04:49+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=7d69ff3ddf0fb83c6a1ec02f85b01b454080b647'/>
<id>7d69ff3ddf0fb83c6a1ec02f85b01b454080b647</id>
<content type='text'>
1. Use Arel for composable queries.
2. For a project with ~10k issues, the page loads in around 600ms.
   Previously, a project with ~5k issues would have a ~20s page load
   time.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Use Arel for composable queries.
2. For a project with ~10k issues, the page loads in around 600ms.
   Previously, a project with ~5k issues would have a ~20s page load
   time.
</pre>
</div>
</content>
</entry>
<entry>
<title>Improve performance of the cycle analytics page.</title>
<updated>2016-09-15T09:23:02+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-15T08:59:36+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/gitlab/gitlab-ce.git/commit/?id=ba25e2f1ac61b47940f939a2d9f1d0ad417e1de2'/>
<id>ba25e2f1ac61b47940f939a2d9f1d0ad417e1de2</id>
<content type='text'>
1. These changes bring down page load time for 100 issues from more than
   a minute to about 1.5 seconds.

2. This entire commit is composed of these types of performance
   enhancements:

     - Cache relevant data in `IssueMetrics` wherever possible.
     - Cache relevant data in `MergeRequestMetrics` wherever possible.
     - Preload metrics

3. Given these improvements, we now only need to make 4 SQL calls:

    - Load all issues
    - Load all merge requests
    - Load all metrics for the issues
    - Load all metrics for the merge requests

4. A list of all the data points that are now being pre-calculated:

    a. The first time an issue is mentioned in a commit

      - In `GitPushService`, find all issues mentioned by the given commit
        using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at`
        flag for each of them.

      - There seems to be a (pre-existing) bug here - files (and
        therefore commits) created using the Web CI don't have
        cross-references created, and issues are not closed even when
        the commit title is "Fixes #xx".

    b. The first time a merge request is deployed to production

      When a `Deployment` is created, find all merge requests that
      were merged in before the deployment, and set the
      `first_deployed_to_production_at` flag for each of them.

    c. The start / end time for a merge request pipeline

      Hook into the `Pipeline` state machine. When the `status` moves to
      `running`, find the merge requests whose tip commit matches the
      pipeline, and record the `latest_build_started_at` time for each
      of them. When the `status` moves to `success`, record the
      `latest_build_finished_at` time.

    d. The merge requests that close an issue

      - This was a big cause of the performance problems we were having
        with Cycle Analytics. We need to use `ReferenceExtractor` to make
        this calculation, which is slow when we have to run it on a large
        number of merge requests.

      - When a merge request is created, updated, or refreshed, find the
        issues it closes, and create an instance of
        `MergeRequestsClosingIssues`, which acts as a join model between
        merge requests and issues.

      - If a `MergeRequestsClosingIssues` instance links a merge request
        and an issue, that issue closes that merge request.

5. The `Queries` module was changed into a class, so we can cache the
   results of `issues` and `merge_requests_closing_issues` across
   various cycle analytics stages.

6. The code added in this commit is untested. Tests will be added in the
   next commit.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. These changes bring down page load time for 100 issues from more than
   a minute to about 1.5 seconds.

2. This entire commit is composed of these types of performance
   enhancements:

     - Cache relevant data in `IssueMetrics` wherever possible.
     - Cache relevant data in `MergeRequestMetrics` wherever possible.
     - Preload metrics

3. Given these improvements, we now only need to make 4 SQL calls:

    - Load all issues
    - Load all merge requests
    - Load all metrics for the issues
    - Load all metrics for the merge requests

4. A list of all the data points that are now being pre-calculated:

    a. The first time an issue is mentioned in a commit

      - In `GitPushService`, find all issues mentioned by the given commit
        using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at`
        flag for each of them.

      - There seems to be a (pre-existing) bug here - files (and
        therefore commits) created using the Web CI don't have
        cross-references created, and issues are not closed even when
        the commit title is "Fixes #xx".

    b. The first time a merge request is deployed to production

      When a `Deployment` is created, find all merge requests that
      were merged in before the deployment, and set the
      `first_deployed_to_production_at` flag for each of them.

    c. The start / end time for a merge request pipeline

      Hook into the `Pipeline` state machine. When the `status` moves to
      `running`, find the merge requests whose tip commit matches the
      pipeline, and record the `latest_build_started_at` time for each
      of them. When the `status` moves to `success`, record the
      `latest_build_finished_at` time.

    d. The merge requests that close an issue

      - This was a big cause of the performance problems we were having
        with Cycle Analytics. We need to use `ReferenceExtractor` to make
        this calculation, which is slow when we have to run it on a large
        number of merge requests.

      - When a merge request is created, updated, or refreshed, find the
        issues it closes, and create an instance of
        `MergeRequestsClosingIssues`, which acts as a join model between
        merge requests and issues.

      - If a `MergeRequestsClosingIssues` instance links a merge request
        and an issue, that issue closes that merge request.

5. The `Queries` module was changed into a class, so we can cache the
   results of `issues` and `merge_requests_closing_issues` across
   various cycle analytics stages.

6. The code added in this commit is untested. Tests will be added in the
   next commit.
</pre>
</div>
</content>
</entry>
</feed>
