delta/go-git.git/src/image/jpeg/reader.go, branch dev.debug

image/jpeg: fix extended sequential Huffman table selector (Th).

2017-04-27T03:48:40+00:00

Previously, the package did not distinguish between baseline and
extended sequential images. Both are non-progressive images, but the Th
range differs between the two, as per Annex B of
https://www.w3.org/Graphics/JPEG/itu-t81.pdf

Extended sequential images are often emitted by the Guetzli encoder.

Fixes #19913

Change-Id: I3d0f9e16d5d374ee1c65e3a8fb87519de61cff94
Reviewed-on: https://go-review.googlesource.com/41831
Reviewed-by: David Symonds

image/jpeg: reconstruct progressive images even if incomplete.

2016-03-31T00:33:24+00:00

Fixes #14522.

As I said on that issue:

----
This is a progressive JPEG image. There are two dimensions of
progressivity: spectral selection (variables zs and ze in scan.go,
ranging in [0, 63]) and successive approximation (variables ah and al in
scan.go, ranging in [0, 8), from LSB to MSB, although ah=0 implicitly
means ah=8).

For this particular image, there are three components, and the SOS
markers contain this progression:

zs, ze, ah, al:  0  0 0 0	components: 0, 1, 2
zs, ze, ah, al:  1 63 0 0	components: 1
zs, ze, ah, al:  1 63 0 0	components: 2
zs, ze, ah, al:  1 63 0 2	components: 0
zs, ze, ah, al:  1 10 2 1	components: 0
zs, ze, ah, al: 11 63 2 1	components: 0
zs, ze, ah, al:  1 10 1 0	components: 0

The combination of all of these is complete (i.e. spectra 0 to 63 and
bits 8 exclusive to 0) for components 1 and 2, but it is incomplete for
component 0 (the luma component). In particular, there is no data for
component 0, spectra 11 to 63 and bits 1 exclusive to 0.

The image/jpeg code, as of Go 1.6, waits until both dimensions are
complete before performing the de-quantization, IDCT and copy to an
*image.YCbCr. This is the "if zigEnd != blockSize-1 || al != 0 { ...
continue }" code and associated commentary in scan.go.

Almost all progressive JPEG images end up complete in both dimensions
for all components, but this particular image is incomplete for
component 0, so the Go code never writes anything to the Y values of the
resultant *image.YCbCr, which is why the broken output is so dark (but
still looks recognizable in terms of red and blue hues).

My reading of the ITU T.81 JPEG specification (Annex G) doesn't
explicitly say that this is a valid image, but it also doesn't rule it
out.

In any case, the fix is, for progressive JPEG images, to always
reconstruct the decoded blocks (by performing the de-quantization, IDCT
and copy to an *image.YCbCr), regardless of whether or not they end up
complete. Note that, in Go, the jpeg.Decode function does not return
until the entire image is decoded, so we still only want to reconstruct
each block once, not once per SOS (Start Of Scan) marker.
----

A test image was also added, based on video-001.progressive.jpeg. When
decoding that image, inserting a

println("nComp, zs, ze, ah, al:", nComp, zigStart, zigEnd, ah, al)

into decoder.processSOS in scan.go prints:

nComp, zs, ze, ah, al: 3 0 0 0 1
nComp, zs, ze, ah, al: 1 1 5 0 2
nComp, zs, ze, ah, al: 1 1 63 0 1
nComp, zs, ze, ah, al: 1 1 63 0 1
nComp, zs, ze, ah, al: 1 6 63 0 2
nComp, zs, ze, ah, al: 1 1 63 2 1
nComp, zs, ze, ah, al: 3 0 0 1 0
nComp, zs, ze, ah, al: 1 1 63 1 0
nComp, zs, ze, ah, al: 1 1 63 1 0
nComp, zs, ze, ah, al: 1 1 63 1 0

In other words, video-001.progressive.jpeg contains 10 different scans.
This little program below drops half of them (remembering to keep the
"\xff\xd9" End of Image marker):

----
package main

import (
	"bytes"
	"io/ioutil"
	"log"
)

func main() {
	sos := []byte{0xff, 0xda}
	eoi := []byte{0xff, 0xd9}

	src, err := ioutil.ReadFile("video-001.progressive.jpeg")
	if err != nil {
		log.Fatal(err)
	}
	b := bytes.Split(src, sos)
	println(len(b)) // Prints 11.
	dst := bytes.Join(b[:5], sos)
	dst = append(dst, eoi...)
	if err := ioutil.WriteFile("video-001.progressive.truncated.jpeg", dst, 0666); err != nil {
		log.Fatal(err)
	}
}
----

The video-001.progressive.truncated.jpeg was converted to png via
libjpeg and ImageMagick:

djpeg -nosmooth video-001.progressive.truncated.jpeg > tmp.tga
convert tmp.tga video-001.progressive.truncated.png
rm tmp.tga

Change-Id: I72b20cd4fb6746d36d8d4d587f891fb3bc641f84
Reviewed-on: https://go-review.googlesource.com/21062
Reviewed-by: Rob Pike

image/jpeg: don't unread a byte if we've already taken bits from it.

2015-07-14T06:21:57+00:00

This rolls back most of golang.org/cl/8841, aka 2f98bac310, and makes a
different fix. It keeps the TestTruncatedSOSDataDoesntPanic test
introduced by that other CL, which obviously still passes after this CL.

Fixes #11650, a regression (introduced by cl/8841) from Go 1.4.

The original cl/8841 changed the image/jpeg not to panic on an input
given in #10387. We still do not panic on that input, after this CL.

I have a corpus of over 160,000 JPEG images, a sample of a web crawl.
The image/jpeg code ran happily over that whole corpus both before and
after this CL, although that corpus clearly didn't catch the regression
in the first place.

This code was otherwise tested manually. I don't think that it's trivial
to synthesize a JPEG input that happens to run out of Huffman data at
just the right place. The test image attached to #11650 obviously has
that property, but I don't think we can simply add that test image to
the repository: it's 227KiB, and I don't know its copyright status.

I also looked back over the issue tracker for problematic JPEGs that
people have filed. The Go code, after this CL, is still happy on these
files in my directory:
issue2362a.jpeg
issue3916.jpeg
issue3976.jpeg
issue4084.jpeg
issue4259.jpeg
issue4291.jpeg
issue4337.jpeg
issue4500.jpeg
issue4705.jpeg
issue4975.jpeg
issue5112.jpeg
issue6767.jpeg
issue9888.jpeg
issue10133.jpeg
issue10357.jpeg
issue10447.jpeg
issue11648.jpeg
issue11650.jpeg

There were other images attached in the issue tracker that aren't
actually valid JPEGs. They failed both before and after this CL:
broken-issue2362b.jpeg
broken-issue6450.jpeg
broken-issue8693.jpeg
broken-issue10154.jpeg
broken-issue10387.jpeg
broken-issue10388.jpeg
broken-issue10389.jpeg
broken-issue10413.jpeg

In summary, this CL fixes #11650 and, after some automated and manual
testing, I don't think introduces new regressions.

Change-Id: I30b67036e9b087f3051d57dac7ea05fb4fa36f66
Reviewed-on: https://go-review.googlesource.com/12163
Reviewed-by: Rob Pike

all: switch to the new deprecation convention

2015-06-18T19:16:23+00:00

While we're at it, move some misplaced comment blocks around.

Change-Id: I1847d7f1ca1dbb8e5de737203c4ed6c66e112508
Reviewed-on: https://go-review.googlesource.com/10188
Reviewed-by: Rob Pike 
Reviewed-by: Russ Cox

image/jpeg: ensure that we can't unread a byte if we didn't read a byte.

2015-04-22T00:20:17+00:00

Fixes #10413

Change-Id: I7a4ecd042c40f786ea7406c670d561b1c1179bf0
Reviewed-on: https://go-review.googlesource.com/8998
Reviewed-by: Rob Pike

image/jpeg: don't assume that an ensureNBits failure implies that we can

2015-04-14T07:22:44+00:00

call unreadByteStuffedByte.

If ensureNBits was due to an io.EOF that was translated to
jpeg.errShortHuffmanData, then we may have read no bytes, so there is no
byte-stuffed-byte to unread.

Fixes #10387

Change-Id: I39a3842590c6cef2aa48943288d52f603338b44d
Reviewed-on: https://go-review.googlesource.com/8841
Reviewed-by: Rob Pike

image/jpeg: reject multiple Start-Of-Frame markers.

2015-04-09T02:32:23+00:00

Fixes #10389

Change-Id: Id1c687122751f9317041d9e425d03b267a26c6de
Reviewed-on: https://go-review.googlesource.com/8681
Reviewed-by: Rob Pike

image/internal/imageutil: new package, used by image/draw and image/jpeg.

2015-03-23T23:22:58+00:00

The imageutil.DrawYCbCr function lives in an internal package because it
is needed by both the image/draw and image/jpeg packages, but it doesn't
seem right for one of those two to depend on the other.

It could eventually go into the image package, but that would require
committing to an API for the rest of Go 1.x.

Change-Id: I7b12555c970d86409365e99eef9360702aaffa30
Reviewed-on: https://go-review.googlesource.com/7925
Reviewed-by: Brad Fitzpatrick 
Reviewed-by: Rob Pike

image/jpeg: reject bad Tq values in SOF data.

2015-03-13T05:22:55+00:00

Fixes #10154

Change-Id: Ibb8ea9bcf512e7639c57a6f17afbe4495fa329cd
Reviewed-on: https://go-review.googlesource.com/7494
Reviewed-by: Minux Ma

image/jpeg: support chroma hv values other than 0x11.

2015-03-11T00:10:09+00:00

The testdata was generated by:
convert video-001.png tmp1.tga
cjpeg -quality 100 -sample 2x2,1x2,1x2 tmp1.tga > video-001.221212.jpeg
djpeg -nosmooth -targa video-001.221212.jpeg > tmp2.tga
convert tmp2.tga video-001.221212.png
rm tmp1.tga tmp2.tga

Change-Id: Ica241dfc19b3eb47ade150bf0432373c6006c38a
Reviewed-on: https://go-review.googlesource.com/7264
Reviewed-by: Rob Pike