Provide `git_iterator_walk` to walk each iterator in lockstep, returning each iterator's idea of the contents of the next path.
When a file on the workdir has the same or a newer timestamp than the index, we need to perform a full check of the contents, as the update of the file may have happened just after we wrote the index. The iterator changes are such that we can reach inside the workdir iterator from the diff, though it may be better to have an accessor instead of moving these structs into the header.
We cannot know from looking at .gitmodules whether a directory is a submodule or not. We need the index or tree we are comparing against to tell us. Otherwise we have to assume the entry in .gitmodules is stale or otherwise invalid. Thus we pass the index of the repository into the workdir iterator, even if we do not want to compare against it. This follows what git does, which even for `git diff <tree>`, it will consider staged submodules as such.
The diff code was using an "ignored_prefix" directory to track if a parent directory was ignored that contained untracked files alongside tracked files. Unfortunately, when negative ignore rules were used for directories inside ignored parents, the wrong rules were applied to untracked files inside the negatively ignored child directories. This commit moves the logic for ignore containment into the workdir iterator (which is a better place for it), so the ignored-ness of a directory is contained in the frame stack during traversal. This allows a child directory to override with a negative ignore and yet still restore the ignored state of the parent when we traverse out of the child. Along with this, there are some problems with "directory only" ignore rules on container directories. Given "a/*" and "!a/b/c/" (where the second rule is a directory rule but the first rule is just a generic prefix rule), then the directory only constraint was having "a/b/c/d/file" match the first rule and not the second. This was fixed by having ignore directory-only rules test a rule against the prefix of a file with LEADINGDIR enabled. Lastly, spot checks for ignores using `git_ignore_path_is_ignored` were tested from the top directory down to the bottom to deal with the containment problem, but this is wrong. We have to test bottom to top so that negative subdirectory rules will be checked before parent ignore rules. This does change the behavior of some existing tests, but it seems only to bring us more in line with core Git, so I think those changes are acceptable.
This adds an option to refresh the stat cache while generating status. It also rips out the GIT_PERF stuff I had an makes use of the trace API to keep statistics about what happens during diff.
In the iterator, distinguish between ignores and empty directories so that diff and status can ignore empty directories, but checkout and stash can treat them as untracked items.
When diff finds an untracked directory, it emulates Git behavior by looking inside the directory to see if there are any untracked items inside it. If there are only ignored items inside the dir, then diff considers it ignored, even if there is no direct ignore rule for it. Checkout was not copying this behavior - when it found an untracked directory, it just treated it as untracked. Unfortunately, when combined with GIT_CHECKOUT_REMOVE_UNTRACKED, this made is seem that checkout (and stash, which uses checkout) was removing ignored items when you had only asked it to remove untracked ones. This commit moves the logic for advancing past an untracked dir while scanning for non-ignored items into an iterator helper fn, and uses that for both diff and checkout.