summaryrefslogtreecommitdiff
path: root/ext/mbstring/tests
Commit message (Collapse)AuthorAgeFilesLines
* Remove stray mentions of mbstring.func_overloadMax Semenik2021-02-154-4/+0
| | | | | | This feature has been completely removed. Closes GH-6688.
* Deprecate passing null to non-nullable arg of internal functionNikita Popov2021-02-115-66/+27
| | | | | | | | | | | | | | | | | | | | | This deprecates passing null to non-nullable scale arguments of internal functions, with the eventual goal of making the behavior consistent with userland functions, where null is never accepted for non-nullable arguments. This change is expected to cause quite a lot of fallout. In most cases, calling code should be adjusted to avoid passing null. In some cases, PHP should be adjusted to make some function arguments nullable. I have already fixed a number of functions before landing this, but feel free to file a bug if you encounter a function that doesn't accept null, but probably should. (The rule of thumb for this to be applicable is that the function must have special behavior for 0 or "", which is distinct from the natural behavior of the parameter.) RFC: https://wiki.php.net/rfc/deprecate_null_to_scalar_internal_arg Closes GH-6475.
* Update 'East Asian Width' table to comply with Unicode 13.0Alex Dowad2021-01-191-25/+0
| | | | | | | | | | | | | | | Instead of manually maintaining the data in eaw_table.h, it is now automatically generated by ucgendat/ucgendat.php, using the EastAsianWidth.txt file from the Unicode Consortium. Something must be said about the deleted test case. Back in 2004, someone noticed that `mb_strwidth` didn't comply with Unicode 4.0. A test case was added to expose the problem. Well, time keeps moving on, and with the changing years, new Unicodes are born and old Unicodes die. Some characters which were counted as double-width in Unicode 4.0 are no longer such in Unicode 13.0, which renders the test case obsolete. At the same time, make a couple of spelling/grammar fixes in ucgendat.php.
* CP5022{0,1,2}: treat truncated multibyte characters as errorAlex Dowad2021-01-151-0/+7
|
* Add test suite for CP5022{0,1,2}Alex Dowad2021-01-151-0/+286
|
* Print "interned" instead of fake refcount in debug_zval_dump()Nikita Popov2021-01-151-41/+41
| | | | | | | | | debug_zval_dump() currently prints refcount 1 for interned strings and arrays, which does not really reflect the truth. These values are not refcounted, so the refcount is misleading. Instead print an "interned" tag. Closes GH-6598.
* JIS7/JIS8 encoding: treat truncated multibyte characters as errorAlex Dowad2021-01-141-0/+11
|
* JIS7/JIS8 encoding: handle invalid 2nd byte for Kanji correctlyAlex Dowad2021-01-143-0/+6527
| | | | | | | | | | | | | | | | Previously, in ISO-2022-JP/JIS7/JIS8, if an escape sequence (starting with 0x1B) appeared where the 2nd byte of a multibyte character should have been, mbstring would forget all about the truncated multibyte character and happily accept the escape sequence. However, such sequences are not legal and should be flagged as errors. Also, any other illegal bytes appearing where the 2nd byte of a multibyte character was expected were just passed through quietly to the output. Fix that. Also add a test suite for both ISO-2022-JP and JIS7/JIS8. (These are extremely similar encodings; JIS7 and JIS8 are variants of ISO-2022-JP. mbstring's 'JIS' is actually a combination of JIS7 _and_ JIS8, since the extensions which each one adds to ISO-2022-JP are disjoint.)
* ISO-2022-JP-2004 conversion: handle invalid characters correctlyAlex Dowad2021-01-143-0/+18662
|
* Convert U+00AF (MACRON) to 0x8150 (FULLWIDTH MACRON) in some SJIS variantsAlex Dowad2020-11-254-0/+9
| | | | | | Except for vanilla Shift-JIS, where 0x7E is a halfwidth overline/macron. As for Shift-JIS-2004, it has an added character (byte sequence 0x854A) which was defined as a halfwidth macron in JIS X 0213:2000, so we use that.
* Convert U+FF5E (FULLWIDTH TILDE) to 0x8160 (WAVE DASH) in SJIS variantsAlex Dowad2020-11-251-0/+3
| | | | | | | | By entering this character in the JIS X 0208 conversion table, we can remove a bunch of explicit `if` clauses in different conversion filters. It also means that U+FF5E can be converted into SJIS-mac now; I don't know why this one SJIS variant rejected U+FF5E before, since 0x8160 means the same thing in SJIS-mac as the others.
* Convert U+203E (OVERLINE) to 0x8150 (FULLWIDTH MACRON) in some SJIS variantsAlex Dowad2020-11-254-0/+11
| | | | | | | | | | Converting U+203E to 0x7E was especially wrong for CP932, where 0x7E represents a tilde. For vanilla Shift-JIS and Shift-JIS-2004, converting to 0x7E is acceptable, since 0x7E does represent an overline/macron in those encodings. Follow the same principle in CP51932, which is closely related to CP932.
* 0x7E is not a tilde in Shift-JIS{,-2004}Alex Dowad2020-11-252-4/+5
|
* 0x5C is not a Yen sign in CP932 (or CP51932)Alex Dowad2020-11-252-0/+5
| | | | | | | | | | | | | | When Microsoft created CP932 (their version of Shift-JIS), they explicitly used bytes 0-0x7F to represent ASCII characters rather than JIS X 0201 characters. So when converting Unicode to CP932, it is not correct to convert U+00A5 to CP932 0x5C. Fortunately, CP932 does have a multi-byte FULLWIDTH YEN SIGN character which we can use instead. CP51932 uses the same extended character set as CP932; while CP932 is MicroSoft's extended version of Shift-JIS, CP51932 is their extended version of EUC-JP. So the same reasoning applies to CP51932.
* 0x5C is not a backslash in Shift-JIS-2004Alex Dowad2020-11-251-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Shift-JIS-2004 is an extension of Shift-JIS, which uses 0x5C for the Yen sign. Therefore, it is not correct to convert ASCII 0x5C (backslash) to Shift-JIS-2004 0x5C (yen sign). JIS X 0208 does have a backslash, so we can convert ASCII backslash to SJIS-2004 backslash instead. From time immemorial, there has been confusion around the treatment of 0x5C bytes on systems using legacy Japanese encodings. JIS X 0201 specified that 0x5C means a yen sign, and thus fonts on Japanese systems, including early versions of Windows, displayed a 0x5C byte as a yen sign. This meant that when ASCII text files were displayed on such systems, what were meant to be backslashes would appear as yen signs. Japanese C programmers could write character escapes using yen signs, and C compilers built on the assumption that the input was ASCII would interpret these escapes as desired. Likewise for shell scripts. Et cetera, et cetera... Therefore, if the input to `mb_convert_encoding` is (for example) a C program, and after converting to Shift-JIS-2004, the user wishes to feed the output into a C compiler, *then* perhaps ASCII 0x5C should be mapped to SJIS 0x5C. However, this scenario is ridiculous and will never happen. A more realistic scenario might be: an article written in SJIS-2004 has embedded Windows file paths (like 'C:\Program Files'), with yen signs used as a path separator. If we convert SJIS-2004 0x5C to ASCII 0x5C, then the path separators will be 'fixed' by the conversion. For general written texts, it is much better to convert backslashes to... backslashes. And yen signs, to yen signs.
* Enhance handling of CP51932 encodingAlex Dowad2020-11-252-0/+7717
| | | | | - Don't pass 'control' characters through in the middle of a multi-byte char - Treat truncated multi-byte characters as an error
* Fix mbstring support for SJIS-Mobile (DoCoMo, KDDI, and Softbank variants of ↵Alex Dowad2020-11-253-0/+1059
| | | | | | | | | | | | | | | | | | | | | | | | | | Shift-JIS) Lots of problems here. - Don't pass 'control' characters through silently in the middle of a multi-byte character. - Treat it as an error if a multi-byte character is truncated. - For ESC sequences used to encode emoji on earlier Softbank phones, if an invalid ESC sequence is found, don't pass it through. Rather, handle it as an error and respect `mb_substitute_character`. - In ranges used by mobile vendors for emoji, if a certain byte sequence doesn't map to any emoji, don't emit a mangled value (actually a raw (ku*94)+ten value, which may not even be a valid Unicode codepoint at all). - When converting Unicode to SJIS-Mobile, don't mangle codepoints which fall in the 2nd range of MicroSoft vendor extensions. Some vendor-specific emoji have been mapped to standard Unicode codepoints now, rather than 'private use area' codepoints. When the legacy code was written, these codepoints may not have existed yet in the Unicode standard which was current at that time. Also do a major code cleanup -- remove dead code, rearrange what is left, use some new macros and helper functions to make the code clearer...
* Enhance handling of CP932 text encodingAlex Dowad2020-11-252-0/+8098
| | | | | | | - Don't allow control characters to appear in the middle of a multi-byte character. (This was a strange feature of mbstring; it doesn't make much sense, and iconv doesn't allow it.) - Treat truncated multi-byte characters as an error.
* Bugfixes for findInvalidChars (helper for mbstring test suite)Alex Dowad2020-11-251-2/+2
|
* Leading BOM is stripped for UTF-32Alex Dowad2020-11-111-12/+9
| | | | | | For consistency with UTF-16 and UCS-4. Also, do some code cleanup.
* Add test suite for SJIS-mac encodingAlex Dowad2020-11-112-0/+7813
|
* Add test suite for SJIS-2004 encodingAlex Dowad2020-11-112-0/+11611
|
* Test EUC-JP and Shift-JIS more thoroughlyAlex Dowad2020-11-114-12/+13
| | | | | | | Previously, the unit tests for these text encodings covered all mappings from legacy -> Unicode, and all _reversible_ mappings from Unicode -> legacy. However, we should also test the few Unicode -> legacy mappings which are not reversible.
* Remove mbstring identify filtersAlex Dowad2020-11-093-16/+15
| | | | | | | | | | | | | | | | | | | | | | | mbstring had an 'identify filter' for almost every supported text encoding which was used when auto-detecting the most likely encoding for a string. It would run over the string and set a 'flag' if it saw anything which did not appear likely to be the encoding in question. One problem with this scheme was that encodings which merely appeared less likely to be the correct one were completely rejected, even if there was no better candidate. Another problem was that the 'identify filters' had a huge amount of code duplication with the 'conversion filters'. Eliminate the identify filters. Instead, when auto-detecting text encoding, use conversion filters to see whether the input string is valid in candidate encodings or not. At the same type, watch the type of codepoints which the string decodes to and mark it as less likely if non-printable characters (ESC, form feed, bell, etc.) or 'private use area' codepoints are seen. Interestingly, one old test case in which JIS text was misidentified as UTF-8 (and this wrong behavior was enshrined in the test) was 'fixed' and the JIS string is now auto-detected as JIS.
* Fix mbstring support for EUC-JP text encodingAlex Dowad2020-11-092-0/+13214
| | | | | | | | | | | - Don't allow control characters to appear in the middle of a multi-byte character. (A strange feature, or perhaps misfeature, of mbstring which is not present in other libraries such as iconv.) - When checking whether string is valid, reject kuten codes which do not map to any character, whether converting from EUC-JP to another encoding, or converting another encoding which uses JIS X 0208/0212 charsets to EUC-JP. - Truncated multi-byte characters are treated as an error.
* Fix mbstring support for Shift-JISAlex Dowad2020-11-093-525/+7158
| | | | | | | | | | | | | | | | | | | | | | | | | | | - Reject otherwise valid kuten codes which don't map to anything in JIS X 0208. - Handle truncated multi-byte characters as an error. - Convert Shift-JIS 0x7E to Unicode 0x203E (overline) as recommended by the Unicode Consortium, and as iconv does. - Convert Shift-JIS 0x5C to Unicode 0xA5 (yen sign) as recommended by the Unicode Consortium, and as iconv does. (NOTE: This will affect PHP scripts which use an internal encoding of Shift-JIS! PHP assigns a special meaning to 0x5C, the backslash. For example, it is used for escapes in double-quoted strings. Mapping the Shift-JIS yen sign to the Unicode yen sign means the yen sign will not be usable for C escapes in double-quoted strings. Japanese PHP programmers who want to write their source code in Shift-JIS for some strange reason will have to use the JIS X 0208 backlash or 'REVERSE SOLIDUS' character for their C escapes.) - Convert Unicode 0x5C (backslash) to Shift-JIS 0x815F (reverse solidus). - Immediately handle error if first Shift-JIS byte is over 0xEF, rather than waiting to see the next byte. (Previously, the value used was 0xFC, which is the limit for the 2nd byte and not the 1st byte of a multi-byte character.) - Don't allow 'control characters' to appear in the middle of a multi-byte character. The test case for bug 47399 is now obsolete. That test assumed that a number of Shift-JIS byte sequences which don't map to any character were 'valid' (because the byte values were within the legal ranges).
* Remove useless byte{2,4}{be,le} encodings from mbstringAlex Dowad2020-11-091-40/+16
| | | | | | | | There is no meaningful difference between these and UCS-{2,4}. They are just a little bit more lax about passing errors silently. They also have no known use. Alias to UCS-{2,4} in case someone, somewhere is using them.
* Fix issues with mbstring encoding testsAlex Dowad2020-11-092-8/+22
| | | | | I made some mistakes on this code, which meant that not everything which should be tested was actually being tested.
* Add test suite for ARMSCII-8 encodingAlex Dowad2020-11-022-0/+288
|
* Add test suite for KOI8-U encodingAlex Dowad2020-11-022-0/+316
|
* Add test suite for KOI8-R encodingAlex Dowad2020-11-022-0/+309
|
* Add test suite for CP850 encodingAlex Dowad2020-11-022-0/+287
|
* Add test suite for CP866 encodingAlex Dowad2020-11-022-0/+288
|
* Add test suite for CP1254 encodingAlex Dowad2020-11-022-0/+289
|
* Add test suite for CP1251 encodingAlex Dowad2020-11-022-0/+289
|
* Test cases for mbstring encodings are less repetitiveAlex Dowad2020-11-023-41/+48
|
* Add test suite for CP1252 encodingAlex Dowad2020-10-305-60/+416
| | | | | | | | | | | | Also remove a bogus test (bug62545.phpt) which wrongly assumed that all invalid characters in CP1251 and CP1252 should map to Unicode 0xFFFD (REPLACEMENT CHARACTER). mbstring has an interface to specify what invalid characters should be replaced with; it's called `mb_substitute_character`. If a user wants to see the Unicode 'replacement character', they can specify that using `mb_substitute_character`. But if they specify something else, we should follow that.
* Add test suite for ISO-8859-x encoding verification and conversionAlex Dowad2020-10-1617-0/+4474
|
* Merge branch 'PHP-8.0'Nikita Popov2020-10-1312-65/+65
|\ | | | | | | | | * PHP-8.0: Normalize mb_ereg() return value
| * Normalize mb_ereg() return valueNikita Popov2020-10-1312-65/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | mb_ereg()/mb_eregi() currently have an inconsistent return value based on whether the $matches parameter is passed or not: > Returns the byte length of the matched string if a match for > pattern was found in string, or FALSE if no matches were found > or an error occurred. > > If the optional parameter regs was not passed or the length of > the matched string is 0, this function returns 1. Coupling this behavior to the $matches parameter doesn't make sense -- we know the match length either way, there is no technical reason to distinguish them. However, returning the match length is not particularly useful either, especially due to the need to convert 0-length into 1-length to satisfy "truthy" checks. We could always return 1, which would kind of match the behavior of preg_match() -- however, preg_match() actually returns the number of matches, which is 0 or 1 for preg_match(), while false signals an error. However, mb_ereg() returns false both for no match and for an error. This would result in an odd 1|false return value. The patch canonicalizes mb_ereg() to always return a boolean, where true indicates a match and false indicates no match or error. This also matches the behavior of the mb_ereg_match() and mb_ereg_search() functions. This fixes the default value integrity violation in PHP 8. Closes GH-6331.
* | Add identify filter for UTF-16, UTF-16LE, UTF-16BEAlex Dowad2020-10-131-0/+2
|/ | | | | There was one faulty test in the suite which only passed before because UTF-16 had no identify filter. After this was fixed, it exposed the problem with the test.
* Update mbstring parameter namesNikita Popov2020-09-2812-21/+21
| | | | Closes GH-6207.
* Standardize mbstring and string on using 'string' as a parameter name.Larry Garfield2020-09-211-1/+1
| | | | Closes GH-6171.
* Consolidate the usage of "either" and "one of" in error messagesMáté Kocsis2020-09-202-2/+2
| | | | Closes GH-6173
* Run tidyNikita Popov2020-09-189-21/+21
| | | | | This should fix most of the remaining issues with tabs and spaces being mixed in tests.
* Promote a few remaining errors in ext/standardMáté Kocsis2020-09-152-2/+3
| | | | Closes GH-6110
* Throw ValueError on null bytes in mb_send_mail()Nikita Popov2020-09-111-0/+38
| | | | Instead of silently replacing with spaces.
* Promote some warnings in MBString RegexesGeorge Peter Banyard2020-09-099-118/+117
| | | | Closes GH-5341
* Throw on invalid mb_http_input() typeNikita Popov2020-09-071-0/+17
|
* Handle null encoding in mb_http_input()Nikita Popov2020-09-041-0/+11
|