This fixes an issue where in code like this:
```
int foo(float bar) // hello() {}
```
The lexer would match `(float bar) // hello()`
as the parameters of the function `foo`, instead
of just `(float bar)`.
In addition, a similar test case to what was originally
reported in #2208 is added.
Something like
id id2("){ ... }");
is no longer wrongly recognized as a "function"
id id2(") {
...
}
");
As the difference in the tests shows, this has the unfortunate side
effect that we no longer highlight something like
int f(param="default");
as a function declaration, but it is hard to imagine another way to
fix this (cf. “most vexing parse” problem).
Fixes #2207
Detect function with return types of more than a single word length (like `unsigned int` or `long long`).
* Fix #1237 cpp whitespace token usage expanded * Adapt tests change to 3eff56f5
The CFamilyLexer was matching whitespace as Text instead of Whitespace.
CFamilyLexer failed to tokenize preprocessor macros when they were preceded by line break surrounded by spaces. This was the case because prerpocessor regex rule expected to start at the beginning of the line, but the space regex rule matched also the whitespace after the line break. Now the space rule has been refined not to match the line break. Because of this, the preprocessor regex rule correctly matches prerpocessor tokens even when they are preceded by white spaces, at the cost of adding some more tokens in the token stream in some cases. This change preserves the behavior of invalid preprocessor usage failing to tokenize.
* Remove duplicate lines in state * Refactor ident regex for readability Refactor ident regex changed in c1a0d82 to improve readability. Refactor namespaced_ident regex added in c1a0d82 to improve readability. * Fix inline keword lexing * Fix indentation * Refactor CLexer keywords to 'keywords' state
* Fix lexing of function names This fixes #1561. Add a keywords state that matches inside and outside functions for keywords. Before this, when a keyword would appear the lexer would go to the statements state, in which functions were not matched. * Add tests for lexing of function names * Unbreak previous tests * Allow namespaced names in function statements Add a second identifiers regex that matces all the previous identifiers and also '::'. I took the decision to create a second identifiers regex with '::' inside, simply because using the old identifiers regex would hurt performance massively on every solution I tried to craft. * Add tests for namespaced names in functions * Unbreak previous tests * Add support for namespaces in namespace declarations Add a namespace state that is entered each time the namespace keyword is matched and lexes all name matches as namespaces. Cases this approach doesn't cover: * Namespaces in using declarations. * Namespaces that prefix names in random code. Unfortunately, in both of these cases the names before and after '::' are not always namespaces. * Add tests for namespace declartions * Unbreak previous tests * Tidy functions regex Remove group nesting that became unneeded after fc56ab8 (the last big refactor). * Remove f string usage I introduced by mistake
They are ok in small snippets to demonstrate error cases. Also recode all examplefiles to UTF-8.
Co-authored-by: Georg Brandl <georg@python.org>