* NF: adding an example of use of simple filter
@simplefilter is great, but also not very intuitive. Indeeds, the syntax seems
to indicate that you define a function with four arguments while in reality you
define a class whose constructor takes arbitrary keyword arguments. I believe in
this case an example to show how to instantiate this filter is really necessary.
Regarding simplefilter, I also believe that it could be improved in two simple
ways:
* accepting any method which takes lexer and stream as a filter. That would be
sufficient as long as there is no option
* the @simplefilter decorator could deal with `self` so that the user do not
have to add it themselves. Probably not worth doing it no, as it would break
compatibility with current version, but would be even simpler to use
* NF: clarifying get_..._options
get_bool_opt's documentation seems to indicate that the key is interpreted as a
Boolean. While a quick look at the code shows clearly that the value associated
to the key is what is interpreted as a Boolean. I hope I made the code clearer
to any people who know python by indicating that it is essentially `.get` but
with extra features
* NF: clarifying Filter
`filter` has already a specific behavior in general python, or for any people
used to functional programing (and even if some dom processor). So indicating
that a filter is not something that remove some tokens seems really useful to
try to explain what is going on.
* NF: adding details regarding states in lexer
I found the state explanation confusing. I do know what a state machine
is. However, reading the code, I first thought that there were two distinct
variables:
* the current state
* the stack
that are somehow related but distinct. Explaining that the current state is the
top of the stack was lacking in my opinion. That also help explain #push. In
particular that if you define in state "s" an operation whose new state is
"#push", the behavior can be quite different than if the new state was "s".