<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/libgit2.git/tests/merge/trees, branch ethomson/test_https</title>
<subtitle>github.com: libgit2/libgit2.git
</subtitle>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/'/>
<entry>
<title>str: introduce `git_str` for internal, `git_buf` is external</title>
<updated>2021-10-17T13:49:01+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2021-09-07T21:53:49+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=f0e693b18afbe1de37d7da5b5a8967b6c87d8e53'/>
<id>f0e693b18afbe1de37d7da5b5a8967b6c87d8e53</id>
<content type='text'>
libgit2 has two distinct requirements that were previously solved by
`git_buf`.  We require:

1. A general purpose string class that provides a number of utility APIs
   for manipulating data (eg, concatenating, truncating, etc).
2. A structure that we can use to return strings to callers that they
   can take ownership of.

By using a single class (`git_buf`) for both of these purposes, we have
confused the API to the point that refactorings are difficult and
reasoning about correctness is also difficult.

Move the utility class `git_buf` to be called `git_str`: this represents
its general purpose, as an internal string buffer class.  The name also
is an homage to Junio Hamano ("gitstr").

The public API remains `git_buf`, and has a much smaller footprint.  It
is generally only used as an "out" param with strict requirements that
follow the documentation.  (Exceptions exist for some legacy APIs to
avoid breaking callers unnecessarily.)

Utility functions exist to convert a user-specified `git_buf` to a
`git_str` so that we can call internal functions, then converting it
back again.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
libgit2 has two distinct requirements that were previously solved by
`git_buf`.  We require:

1. A general purpose string class that provides a number of utility APIs
   for manipulating data (eg, concatenating, truncating, etc).
2. A structure that we can use to return strings to callers that they
   can take ownership of.

By using a single class (`git_buf`) for both of these purposes, we have
confused the API to the point that refactorings are difficult and
reasoning about correctness is also difficult.

Move the utility class `git_buf` to be called `git_str`: this represents
its general purpose, as an internal string buffer class.  The name also
is an homage to Junio Hamano ("gitstr").

The public API remains `git_buf`, and has a much smaller footprint.  It
is generally only used as an "out" param with strict requirements that
follow the documentation.  (Exceptions exist for some legacy APIs to
avoid breaking callers unnecessarily.)

Utility functions exist to convert a user-specified `git_buf` to a
`git_str` so that we can call internal functions, then converting it
back again.
</pre>
</div>
</content>
</entry>
<entry>
<title>tests: merge: fix printf formatter on 32 bit arches</title>
<updated>2020-05-12T09:41:44+00:00</updated>
<author>
<name>Patrick Steinhardt</name>
<email>ps@pks.im</email>
</author>
<published>2020-05-12T09:41:44+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=0cf9b6668c8ac5021cb36c6458c3ed243e6251bd'/>
<id>0cf9b6668c8ac5021cb36c6458c3ed243e6251bd</id>
<content type='text'>
We currently use `PRIuMAX` to print an integer of type `size_t` in
merge::trees::rename::cache_recomputation. While this works just fine on
64 bit arches, it doesn't on 32 bit ones. As a result, our nightly
builds on x86 and arm32 fail.

Fix the issue by using `PRIuZ` instead.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
We currently use `PRIuMAX` to print an integer of type `size_t` in
merge::trees::rename::cache_recomputation. While this works just fine on
64 bit arches, it doesn't on 32 bit ones. As a result, our nightly
builds on x86 and arm32 fail.

Fix the issue by using `PRIuZ` instead.
</pre>
</div>
</content>
</entry>
<entry>
<title>merge: cache negative cache results for similarity metrics</title>
<updated>2020-04-01T13:16:18+00:00</updated>
<author>
<name>Patrick Steinhardt</name>
<email>ps@pks.im</email>
</author>
<published>2020-04-01T13:16:18+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=4dfcc50fe88e06a49a4ccccb048d47fe6be01292'/>
<id>4dfcc50fe88e06a49a4ccccb048d47fe6be01292</id>
<content type='text'>
When computing renames, we cache the hash signatures for each of the
potentially conflicting entries so that we do not need to repeatedly
read the file and can at least halfway efficiently determine whether two
files are similar enough to be deemed a rename. In order to make the
hash signatures meaningful, we require at least four lines of data to be
present, resulting in at least four different hashes that can be
compared. Files that are deemed too small are not cached at all and
will thus be repeatedly re-hashed, which is usually not a huge issue.

The issue with above heuristic is in case a file does _not_ have at
least four lines, where a line is anything separated by a consecutive
run of "\n" or "\0" characters. For example "a\nb" is two lines, but
"a\0\0b" is also just two lines. Taken to the extreme, a file that has
megabytes of consecutive space- or NUL-only may also be deemed as too
small and thus not get cached. As a result, we will repeatedly load its
blob, calculate its hash signature just to finally throw it away as we
notice it's not of any value. When you've got a comparitively big file
that you compare against a big set of potentially renamed files, then
the cost simply expodes.

The issue can be trivially fixed by introducing negative cache entries.
Whenever we determine that a given blob does not have a meaningful
representation via a hash signature, we store this negative cache marker
and will from then on not hash it again, but also ignore it as a
potential rename target. This should help the "normal" case already
where you have a lot of small files as rename candidates, but in the
above scenario it's savings are extraordinarily high.

To verify we do not hit the issue anymore with described solution, this
commit adds a test that uses the exact same setup described above with
one 50 megabyte blob of '\0' characters and 1000 other files that get
renamed. Without the negative cache:

$ time ./libgit2_clar -smerge::trees::renames::cache_recomputation &gt;/dev/null
real    11m48.377s
user    11m11.576s
sys     0m35.187s

And with the negative cache:

$ time ./libgit2_clar -smerge::trees::renames::cache_recomputation &gt;/dev/null
real    0m1.972s
user    0m1.851s
sys     0m0.118s

So this represents a ~350-fold performance improvement, but it obviously
depends on how many files you have and how big the blob is. The test
number were chosen in a way that one will immediately notice as soon as
the bug resurfaces.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
When computing renames, we cache the hash signatures for each of the
potentially conflicting entries so that we do not need to repeatedly
read the file and can at least halfway efficiently determine whether two
files are similar enough to be deemed a rename. In order to make the
hash signatures meaningful, we require at least four lines of data to be
present, resulting in at least four different hashes that can be
compared. Files that are deemed too small are not cached at all and
will thus be repeatedly re-hashed, which is usually not a huge issue.

The issue with above heuristic is in case a file does _not_ have at
least four lines, where a line is anything separated by a consecutive
run of "\n" or "\0" characters. For example "a\nb" is two lines, but
"a\0\0b" is also just two lines. Taken to the extreme, a file that has
megabytes of consecutive space- or NUL-only may also be deemed as too
small and thus not get cached. As a result, we will repeatedly load its
blob, calculate its hash signature just to finally throw it away as we
notice it's not of any value. When you've got a comparitively big file
that you compare against a big set of potentially renamed files, then
the cost simply expodes.

The issue can be trivially fixed by introducing negative cache entries.
Whenever we determine that a given blob does not have a meaningful
representation via a hash signature, we store this negative cache marker
and will from then on not hash it again, but also ignore it as a
potential rename target. This should help the "normal" case already
where you have a lot of small files as rename candidates, but in the
above scenario it's savings are extraordinarily high.

To verify we do not hit the issue anymore with described solution, this
commit adds a test that uses the exact same setup described above with
one 50 megabyte blob of '\0' characters and 1000 other files that get
renamed. Without the negative cache:

$ time ./libgit2_clar -smerge::trees::renames::cache_recomputation &gt;/dev/null
real    11m48.377s
user    11m11.576s
sys     0m35.187s

And with the negative cache:

$ time ./libgit2_clar -smerge::trees::renames::cache_recomputation &gt;/dev/null
real    0m1.972s
user    0m1.851s
sys     0m0.118s

So this represents a ~350-fold performance improvement, but it obviously
depends on how many files you have and how big the blob is. The test
number were chosen in a way that one will immediately notice as soon as
the bug resurfaces.
</pre>
</div>
</content>
</entry>
<entry>
<title>fileops: rename to "futils.h" to match function signatures</title>
<updated>2019-07-20T17:11:20+00:00</updated>
<author>
<name>Patrick Steinhardt</name>
<email>ps@pks.im</email>
</author>
<published>2019-06-29T07:17:32+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=e54343a4024e75dfaa940e652c2c9799d33634b2'/>
<id>e54343a4024e75dfaa940e652c2c9799d33634b2</id>
<content type='text'>
Our file utils functions all have a "futils" prefix, e.g.
`git_futils_touch`. One would thus naturally guess that their
definitions and implementation would live in files "futils.h" and
"futils.c", respectively, but in fact they live in "fileops.h".

Rename the files to match expectations.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Our file utils functions all have a "futils" prefix, e.g.
`git_futils_touch`. One would thus naturally guess that their
definitions and implementation would live in files "futils.h" and
"futils.c", respectively, but in fact they live in "fileops.h".

Rename the files to match expectations.
</pre>
</div>
</content>
</entry>
<entry>
<title>tests: merge::analysis: use variants to deduplicate test suites</title>
<updated>2019-06-13T09:59:50+00:00</updated>
<author>
<name>Patrick Steinhardt</name>
<email>ps@pks.im</email>
</author>
<published>2019-06-13T09:57:16+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=70fae43c6ce804f0669ba68b17fe9087e6bd58f5'/>
<id>70fae43c6ce804f0669ba68b17fe9087e6bd58f5</id>
<content type='text'>
Since commit 394951ad4 (tests: allow for simple data-driven
tests, 2019-06-07), we have the ability to run a given test suite
with multiple variants. Use this new feature to deduplicate the
test suites for merge::{trees,workdir}::analysis into a single
test suite.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Since commit 394951ad4 (tests: allow for simple data-driven
tests, 2019-06-07), we have the ability to run a given test suite
with multiple variants. Use this new feature to deduplicate the
test suites for merge::{trees,workdir}::analysis into a single
test suite.
</pre>
</div>
</content>
</entry>
<entry>
<title>Review fixes:</title>
<updated>2019-06-10T10:22:02+00:00</updated>
<author>
<name>Robert Coup</name>
<email>robert@coup.net.nz</email>
</author>
<published>2019-06-07T14:22:42+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=21ddeabe6cb4a77899a6c7348493a2b45cc82a9c'/>
<id>21ddeabe6cb4a77899a6c7348493a2b45cc82a9c</id>
<content type='text'>
- whitespace -&gt; tabs
- comment style
- improve repo naming in merge/trees/analysis tests.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- whitespace -&gt; tabs
- comment style
- improve repo naming in merge/trees/analysis tests.
</pre>
</div>
</content>
</entry>
<entry>
<title>Refactor testing:</title>
<updated>2019-06-10T10:22:02+00:00</updated>
<author>
<name>Robert Coup</name>
<email>robert@coup.net.nz</email>
</author>
<published>2019-06-06T15:32:09+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=7b27b6cf716429080fbc968d8f573977996ee112'/>
<id>7b27b6cf716429080fbc968d8f573977996ee112</id>
<content type='text'>
- move duplication between merge/trees/ and merge/workdir/ into merge/analysis{.c,.h}
- remove merge-resolve.git resource, open the existing merge-resolve as a bare repo instead.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- move duplication between merge/trees/ and merge/workdir/ into merge/analysis{.c,.h}
- remove merge-resolve.git resource, open the existing merge-resolve as a bare repo instead.
</pre>
</div>
</content>
</entry>
<entry>
<title>merge: add doc header to analysis tests</title>
<updated>2019-06-10T10:22:02+00:00</updated>
<author>
<name>Robert Coup</name>
<email>robert@coup.net.nz</email>
</author>
<published>2019-03-19T22:51:24+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=5427461fb47d22760c0a5a80b6a5f33561e48820'/>
<id>5427461fb47d22760c0a5a80b6a5f33561e48820</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>merge: tests for bare repo merge analysis</title>
<updated>2019-06-10T10:22:02+00:00</updated>
<author>
<name>Robert Coup</name>
<email>robert@coup.net.nz</email>
</author>
<published>2019-03-19T10:43:56+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=1d04f477e5c3c83b6c3985df21fa5fbec600f302'/>
<id>1d04f477e5c3c83b6c3985df21fa5fbec600f302</id>
<content type='text'>
dupe of workdir/analysis.c against a bare repo.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
dupe of workdir/analysis.c against a bare repo.
</pre>
</div>
</content>
</entry>
<entry>
<title>object_type: use new enumeration names</title>
<updated>2018-12-01T11:54:57+00:00</updated>
<author>
<name>Edward Thomson</name>
<email>ethomson@edwardthomson.com</email>
</author>
<published>2018-11-28T14:26:57+00:00</published>
<link rel='alternate' type='text/html' href='http://91.123.203.49/cgit/delta/libgit2.git/commit/?id=168fe39bea3368972a8b1a33d5908e73bc790c18'/>
<id>168fe39bea3368972a8b1a33d5908e73bc790c18</id>
<content type='text'>
Use the new object_type enumeration names within the codebase.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Use the new object_type enumeration names within the codebase.
</pre>
</div>
</content>
</entry>
</feed>
