tree: fix integer overflow when reading unreasonably large filemodes

The `parse_mode` option uses an open-coded octal number parser. The
parser is quite naive in that it simply parses until hitting a character
that is not in the accepted range of '0' - '7', completely ignoring the
fact that we can at most accept a 16 bit unsigned integer as filemode.
If the filemode is bigger than UINT16_MAX, it will thus overflow and
provide an invalid filemode for the object entry.

Fix the issue by using `git__strntol32` instead and doing a bounds
check. As this function already handles overflows, it neatly solves the
problem.

Note that previously, `parse_mode` was also skipping the character
immediately after the filemode. In proper trees, this should be a simple
space, but in fact the parser accepted any character and simply skipped
over it. As a consequence of using `git__strntol32`, we now need to an
explicit check for a trailing whitespace after having parsed the
filemode. Because of the newly introduced error message, the test
object::tree::parse::mode_doesnt_cause_oob_read needs adjustment to its
error message check, which in fact is a good thing as it demonstrates
that we now fail looking for the whitespace immediately following the
filemode.

Add a test that shows that we will fail to parse such invalid filemodes
now.

1 files changed, 7 insertions, 1 deletions

diff --git a/tests/object/tree/parse.c b/tests/object/tree/parse.c
index 67b758667..83d22193d 100644
--- a/tests/object/tree/parse.c
+++ b/tests/object/tree/parse.c
@@ -118,7 +118,13 @@ void test_object_tree_parse__mode_doesnt_cause_oob_read(void)
 	 * later fail to parse the OID with a different error
 	 * message
 	 */
-	cl_assert(strstr(giterr_last()->message, "object is corrupted"));
+	cl_assert_equal_s(giterr_last()->message, "failed to parse tree: missing space after filemode");
+}
+
+void test_object_tree_parse__unreasonably_large_mode_fails(void)
+{
+	const char data[] = "10000000000000000000000000 bar\x00" OID1_HEX;
+	assert_tree_fails(data, ARRAY_SIZE(data) - 1);
 }
 
 void test_object_tree_parse__missing_filename_separator_fails(void)