delta/git.git/utf8.c, branch bc/commit-invalid-utf8

pretty: Fix bug in truncation support for %>, %< and %><

2013-04-28T19:09:37+00:00

Some systems experience failures in t4205-*.sh (tests 18-20, 27)
which all relate to the use of truncation with the %< padding
placeholder. This capability was added in the commit a7f01c6b
("pretty: support truncating in %>, %< and %><", 19-04-2013).

The truncation support was implemented with the assistance of a
new strbuf function (strbuf_utf8_replace). This function contains
the following code:

       strbuf_attach(sb_src, strbuf_detach(&sb_dst, NULL),
                     sb_dst.len, sb_dst.alloc);

Unfortunately, this code is subject to unspecified behaviour. In
particular, the order of evaluation of the argument expressions
(along with the associated side effects) is not specified by the
C standard. Note that the second argument expression is a call to
strbuf_detach() which, as a side effect, sets the 'len' and 'alloc'
fields of the sb_dst argument to zero. Depending on the order of
evaluation of the argument expressions to the strbuf_attach call,
this can lead to assigning an empty string to 'sb_src'.

In order to remove the undesired behaviour, we replace the above
line of code with:

       strbuf_swap(sb_src, &sb_dst);
       strbuf_release(&sb_dst);

which achieves the desired effect without provoking unspecified
behaviour.

Signed-off-by: Ramsay Jones 
Acked-by: Duy Nguyen 
Signed-off-by: Junio C Hamano

pretty: support %>> that steal trailing spaces

2013-04-18T23:28:29+00:00

This is pretty useful in `%<(100)%s%Cred%>(20)% an' where %s does not
use up all 100 columns and %an needs more than 20 columns. By
replacing %>(20) with %>>(20), %an can steal spaces from %s.

%>> understands escape sequences, so %Cred does not stop it from
stealing spaces in %<(100).

Signed-off-by: Nguyễn Thái Ngọc Duy 
Signed-off-by: Junio C Hamano

pretty: support truncating in %>, %< and %><

2013-04-18T23:28:29+00:00

%>(N,trunc) truncates the right part after N columns and replace the
last two letters with "..". ltrunc does the same on the left. mtrunc
cuts the middle out.

Signed-off-by: Nguyễn Thái Ngọc Duy 
Signed-off-by: Junio C Hamano

utf8.c: add reencode_string_len() that can handle NULs in string

2013-04-18T23:28:28+00:00

Signed-off-by: Nguyễn Thái Ngọc Duy 
Signed-off-by: Junio C Hamano

utf8.c: add utf8_strnwidth() with the ability to skip ansi sequences

2013-04-18T23:28:28+00:00

Signed-off-by: Nguyễn Thái Ngọc Duy 
Signed-off-by: Junio C Hamano

utf8.c: move display_mode_esc_sequence_len() for use by other functions

2013-04-18T23:28:27+00:00

Signed-off-by: Nguyễn Thái Ngọc Duy 
Signed-off-by: Junio C Hamano

Merge branch 'ks/rfc2047-one-char-at-a-time'

2013-03-25T21:00:46+00:00

When "format-patch" quoted a non-ascii strings on the header files,
it incorrectly applied rfc2047 and chopped a single character in
the middle of it.

* ks/rfc2047-one-char-at-a-time:
  format-patch: RFC 2047 says multi-octet character may not be split

Merge branch 'jk/utf-8-can-be-spelled-differently'

2013-03-21T21:02:58+00:00

Some platforms and users spell UTF-8 differently; retry with the
most official "UTF-8" when the system does not understand the
user-supplied encoding name that are the common alternative
spellings of UTF-8.

* jk/utf-8-can-be-spelled-differently:
  utf8: accept alternate spellings of UTF-8

format-patch: RFC 2047 says multi-octet character may not be split

2013-03-09T19:11:19+00:00

Even though an earlier attempt (bafc478..41dd00bad) cleaned
up RFC 2047 encoding, pretty.c::add_rfc2047() still decides
where to split the output line by going through the input
one byte at a time, and potentially splits a character in
the middle.  A subject line may end up showing like this:

     ".... fö?? bar".   (instead of  ".... föö bar".)

if split incorrectly.

RFC 2047, section 5 (3) explicitly forbids such beaviour

    Each 'encoded-word' MUST represent an integral number of
    characters.  A multi-octet character may not be split across
    adjacent 'encoded- word's.

that means that e.g. for

    Subject: .... föö bar

encoding

    Subject: =?UTF-8?q?....=20f=C3=B6=C3=B6?=
     =?UTF-8?q?=20bar?=

is correct, and

    Subject: =?UTF-8?q?....=20f=C3=B6=C3?=      <-- NOTE ö is broken here
     =?UTF-8?q?=B6=20bar?=

is not, because "ö" character UTF-8 encoding C3 B6 is split here across
adjacent encoded words.

To fix the problem, make the loop grab one _character_ at a time and
determine its output length to see where to break the output line.  Note
that this version only knows about UTF-8, but the logic to grab one
character is abstracted out in mbs_chrlen() function to make it possible
to extend it to other encodings with the help of iconv in the future.

Signed-off-by: Kirill Smelkov 
Signed-off-by: Junio C Hamano

utf8: accept alternate spellings of UTF-8

2013-02-25T21:17:22+00:00

The iconv implementation on many platforms will accept
variants of UTF-8, including "UTF8", "utf-8", and "utf8",
but some do not. We make allowances in our code to treat
them all identically, but we sometimes hand the string from
the user directly to iconv. In this case, the platform iconv
may or may not work.

There are really four levels of platform iconv support for
these synonyms:

  1. All synonyms understood (e.g., glibc).

  2. Only the official "UTF-8" understood (e.g., Windows).

  3. Official "UTF-8" not understood, but some other synonym
     understood (it's not known whether such a platform exists).

  4. Neither "UTF-8" nor any synonym understood (e.g.,
     ancient systems, or ones without utf8 support
     installed).

This patch teaches git to fall back to using the official
"UTF-8" spelling when iconv_open fails (and the encoding was
one of the synonym spellings). This makes things more
convenient to users of type 2 systems, as they can now use
any of the synonyms for the log output encoding.

Type 1 systems are not affected, as iconv already works on
the first try.

Type 4 systems are not affected, as both attempts already
fail.

Type 3 systems will not benefit from the feature, but
because we only use "UTF-8" as a fallback, they will not be
regressed (i.e., you can continue to use "utf8" if your
platform supports it). We could try all the various
synonyms, but since such systems are not even known to
exist, it's not worth the effort.

Signed-off-by: Jeff King 
Signed-off-by: Junio C Hamano