delta/go-git.git/src/compress, branch dev.typeparams

compress/lzw: add Reset method to Reader and Writer

2021-03-17T07:01:00+00:00

We add a Reset method which clears any internal state of an encoder
or a decoder to let it be reused again as a new Writer or Reader respectively.

We also export the encoder and decoder structs, renaming them
to be Reader and Writer, and we guarantee that the underlying types
from the constructors will always be Reader and Writer respectively.

Benchmark results by reusing the encoder:
on cpu: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz

name                 time/op
Decoder/1e4-8          93.6µs ± 1%
Decoder/1e-Reuse4-8    87.7µs ± 1%
Decoder/1e5-8           877µs ± 1%
Decoder/1e-Reuse5-8     860µs ± 3%
Decoder/1e6-8          8.79ms ± 1%
Decoder/1e-Reuse6-8    8.82ms ± 4%
Encoder/1e4-8           168µs ± 2%
Encoder/1e-Reuse4-8     160µs ± 1%
Encoder/1e5-8          1.64ms ± 1%
Encoder/1e-Reuse5-8    1.61ms ± 2%
Encoder/1e6-8          16.2ms ± 6%
Encoder/1e-Reuse6-8    15.8ms ± 2%

name                 speed
Decoder/1e4-8         107MB/s ± 1%
Decoder/1e-Reuse4-8   114MB/s ± 1%
Decoder/1e5-8         114MB/s ± 1%
Decoder/1e-Reuse5-8   116MB/s ± 3%
Decoder/1e6-8         114MB/s ± 1%
Decoder/1e-Reuse6-8   113MB/s ± 5%
Encoder/1e4-8        59.7MB/s ± 2%
Encoder/1e-Reuse4-8  62.4MB/s ± 1%
Encoder/1e5-8        61.1MB/s ± 1%
Encoder/1e-Reuse5-8  62.0MB/s ± 2%
Encoder/1e6-8        61.7MB/s ± 5%
Encoder/1e-Reuse6-8  63.4MB/s ± 2%

name                 alloc/op
Decoder/1e4-8          21.8kB ± 0%
Decoder/1e-Reuse4-8     50.0B ± 0%
Decoder/1e5-8          21.8kB ± 0%
Decoder/1e-Reuse5-8     70.4B ± 2%
Decoder/1e6-8          21.9kB ± 0%
Decoder/1e-Reuse6-8      271B ± 3%
Encoder/1e4-8          77.9kB ± 0%
Encoder/1e-Reuse4-8    4.17kB ± 0%
Encoder/1e5-8          77.9kB ± 0%
Encoder/1e-Reuse5-8    4.27kB ± 0%
Encoder/1e6-8          77.9kB ± 0%
Encoder/1e-Reuse6-8    5.22kB ± 0%

name                 allocs/op
Decoder/1e4-8            2.00 ± 0%
Decoder/1e-Reuse4-8      1.00 ± 0%
Decoder/1e5-8            2.00 ± 0%
Decoder/1e-Reuse5-8      1.00 ± 0%
Decoder/1e6-8            2.00 ± 0%
Decoder/1e-Reuse6-8      1.00 ± 0%
Encoder/1e4-8            3.00 ± 0%
Encoder/1e-Reuse4-8      2.00 ± 0%
Encoder/1e5-8            3.00 ± 0%
Encoder/1e-Reuse5-8      2.00 ± 0%
Encoder/1e6-8            3.00 ± 0%
Encoder/1e-Reuse6-8      2.00 ± 0%

Fixes #26535

Change-Id: Icde613fea6234a5bdce95f1e49910f5687e30b22
Reviewed-on: https://go-review.googlesource.com/c/go/+/273667
Trust: Agniva De Sarker 
Trust: Joe Tsai 
Reviewed-by: Joe Tsai

all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp

2020-12-09T19:12:23+00:00

As part of #42026, these helpers from io/ioutil were moved to os.
(ioutil.TempFile and TempDir became os.CreateTemp and MkdirTemp.)

Update the Go tree to use the preferred names.

As usual, code compiled with the Go 1.4 bootstrap toolchain
and code vendored from other sources is excluded.

ReadDir changes are in a separate CL, because they are not a
simple search and replace.

For #42026.

Change-Id: If318df0216d57e95ea0c4093b89f65e5b0ababb3
Reviewed-on: https://go-review.googlesource.com/c/go/+/266365
Trust: Russ Cox 
Run-TryBot: Russ Cox 
TryBot-Result: Go Bot 
Reviewed-by: Ian Lance Taylor

all: update to use filepath.WalkDir instead of filepath.Walk

2020-12-02T16:33:57+00:00

Now that filepath.WalkDir is available, it is more efficient
and should be used in place of filepath.Walk.
Update the tree to reflect best practices.

As usual, the code compiled with Go 1.4 during bootstrap is excluded.
(In this CL, that's only cmd/dist.)

For #42027.

Change-Id: Ib0f7b1e43e50b789052f9835a63ced701d8c411c
Reviewed-on: https://go-review.googlesource.com/c/go/+/267719
Trust: Russ Cox 
Run-TryBot: Russ Cox 
TryBot-Result: Go Bot 
Reviewed-by: Rob Pike

all: fix quoting for compress/bzip2 and time's godoc

2020-10-23T21:57:39+00:00

The existing usage of grave accent (`) and apostrophe (')
at some places made godoc to ignore them and show it as it is.
So, use both of the characters twice (consecutively) so that
godoc can convert it to {left,right} double quotation mark.

Fixes #41958

Change-Id: I64fd9b5fa34f416ad595009d09f5482e10bd8b4f
Reviewed-on: https://go-review.googlesource.com/c/go/+/262397
Reviewed-by: Russ Cox 
Trust: Alberto Donizetti

all: update references to symbols moved from io/ioutil to io

2020-10-20T18:41:18+00:00

The old ioutil references are still valid, but update our code
to reflect best practices and get used to the new locations.

Code compiled with the bootstrap toolchain
(cmd/asm, cmd/dist, cmd/compile, debug/elf)
must remain Go 1.4-compatible and is excluded.
Also excluded vendored code.

For #41190.

Change-Id: I6d86f2bf7bc37a9d904b6cee3fe0c7af6d94d5b1
Reviewed-on: https://go-review.googlesource.com/c/go/+/263142
Trust: Russ Cox 
Run-TryBot: Russ Cox 
TryBot-Result: Go Bot 
Reviewed-by: Emmanuel Odeke

all: update references to symbols moved from os to io/fs

2020-10-20T02:32:42+00:00

The old os references are still valid, but update our code
to reflect best practices and get used to the new locations.

Code compiled with the bootstrap toolchain
(cmd/asm, cmd/dist, cmd/compile, debug/elf)
must remain Go 1.4-compatible and is excluded.

For #41190.

Change-Id: I8f9526977867c10a221e2f392f78d7dec073f1bd
Reviewed-on: https://go-review.googlesource.com/c/go/+/243907
Trust: Russ Cox 
Run-TryBot: Russ Cox 
TryBot-Result: Go Bot 
Reviewed-by: Rob Pike

compress/flate: fix corrupted output

2020-10-19T18:48:17+00:00

The fastest compression mode can pick up a false match for every 2GB
of input data resulting in incorrectly decompressed data.

Since matches are allowed to be up to and including at maxMatchOffset
we must offset the buffer by an additional element to prevent the first
4 bytes to match after an out-of-reach value after shiftOffsets has
been called.

We offset by `maxMatchOffset + 1` so offset 0 in the table will now
fail the `if offset > maxMatchOffset` in all cases.

Fixes #41420

Change-Id: If1fbe01728e132b8a207e3f3f439edd832dcc710
GitHub-Last-Rev: 50fabab0da874c37543b139459a810e12e83cee2
GitHub-Pull-Request: golang/go#41477
Reviewed-on: https://go-review.googlesource.com/c/go/+/255879
Reviewed-by: Matthew Dempsky 
Run-TryBot: Matthew Dempsky 
TryBot-Result: Go Bot 
Trust: Joe Tsai 
Trust: Matthew Dempsky

compress/flate: revert a goto for-loop

2020-10-16T14:23:21+00:00

In https://golang.org/cl/16528, a goto loop was chosen over a regular
for loop since that would make the function inlinable.

Thanks to the recent https://golang.org/cl/256459, for loops without a
label can now be inlined. So we can undo the workaround and simplify the
code.

Also add the function to TestIntendedInlining, which passes both before
and after the change, as expected.

For #14768.

Change-Id: Ie5df55a6bcb07c538ca331eef2f908807ff0b516
Reviewed-on: https://go-review.googlesource.com/c/go/+/263037
Trust: Daniel Martí 
Run-TryBot: Daniel Martí 
Reviewed-by: Brad Fitzpatrick 
TryBot-Result: Go Bot

compress/flate: remove unneeded zeroing of bytes array in (*huffmanBitWriter).reset

2020-10-06T20:30:36+00:00

There is no correctness reason to zero out the w.bytes array in (w
*huffmanBitWriter).reset, since w.nbytes is correctly set to zero. The elements of
the bytes array are always written sequentially, with nbytes indicating how many
elements have been written, and are only read up to the current value of nybytes.

We have a pprof profile of a web server that compresses its request/responses, and
the zeroing in reset() is taking up 2.6% of the CPU time of the server (and could
be causing more slowdowns elsewhere due to its effects on the cache). This
overhead may be showing up especially because there are many request/responses
that are all fairly small.

I'm not sure if the zeroing of the bytes array was intended as extra protection of
data across reset uses in the same program, but no protection is needed as long as
the huffman_bit_writer code remains correct.

Change-Id: I67f2b2f56cff9dcc38d8fc0aea885bb010aeedbf
Reviewed-on: https://go-review.googlesource.com/c/go/+/258577
Run-TryBot: Dan Scales 
Run-TryBot: Joe Tsai 
TryBot-Result: Go Bot 
Reviewed-by: Klaus Post 
Reviewed-by: Joe Tsai 
Trust: Joe Tsai 
Trust: Dan Scales

compress/flate: fix another deflate Reset inconsistency

2020-07-16T23:38:32+00:00

While investigating #34121, fixed by CL 193605,
I discovered another case where Reset was not quite
resetting enough.

This specific case is not a problem in Reset itself but
rather that the Huffman bit writer in one code path
is using uninitialized memory left over from a previous
block, making the compression not choose the optimal
compression method.

Fixes #34121.

Change-Id: I29245b28214d924e382f91e2c56b4b8a9b7da13d
Reviewed-on: https://go-review.googlesource.com/c/go/+/243140
Run-TryBot: Russ Cox 
TryBot-Result: Gobot Gobot 
Reviewed-by: Joe Tsai