Large update: Scanners for HTML, RHTML and Nitro-XHTML added.

CSS style changes/enhancments (mainly the new background color for inline code, affects all Ruby code.)
Demos and tests adjusted.

Plugin: new PluginHost::default method.

Scanner:
- New setup method
- ability to re-use a scanner
- ability to keep the tokens
- minor changes to token caching and string flattening

Encoder: Error if token content is neither String nor Symbol.

HTML encoder:
- more warnings for unclosed tokens
- output now UTF-8

Ruby Scanner:
- bug: symbols before => now do not include =; {:foo=>bar} is valid Ruby code
- try to close all open tokens
- constants now all with specific namespace (for speed, I hope)

Styles: new :entity/en class.

Test suite now gives hinted HTML output.

5 files changed, 22 insertions, 4 deletions

diff --git a/test/ruby/besetzung.out.raydebug b/test/ruby/besetzung.out.raydebug
index 861b76d..13f9dc9 100644
--- a/test/ruby/besetzung.out.raydebug
+++ b/test/ruby/besetzung.out.raydebug
diff --git a/test/ruby/evil.out.raydebug b/test/ruby/evil.out.raydebug
index eaa455a..44f381b 100644
--- a/test/ruby/evil.out.raydebug
+++ b/test/ruby/evil.out.raydebug
@@ -442,7 +442,7 @@ ident(p) constant(Foou)operator(.)ident(new)operator(.)operator([]) operator({)i
 

 reserved(if) ident(p) reserved(then) ident(p) reserved(end)

 

-ident(p)operator(()operator({)symbol(:foo=)operator(>)symbol(:bar)operator(})operator(\))   comment(#why does this work? i'd think that ':foo=' would be 1 token)

+ident(p)operator(()operator({)symbol(:foo)operator(=)operator(>)symbol(:bar)operator(})operator(\))   comment(#why does this work? i'd think that ':foo=' would be 1 token)

 ident(p)   constant(EMPTY) operator(=) integer(0)

 ident(p)   constant(BLACK) operator(=) integer(1)

 ident(p)   constant(WHITE) operator(=) operator(-) constant(BLACK)

@@ -666,7 +666,7 @@ end)>
 

 

 ident(proc) operator({)

-  ident(h)operator(=)operator({)symbol(:a=)operator(>)operator(()ident(foo)operator(=)integer(100)operator(\))operator(})

+  ident(h)operator(=)operator({)symbol(:a)operator(=)operator(>)operator(()ident(foo)operator(=)integer(100)operator(\))operator(})

   ident(p)operator(() ident(foo) string<delimiter(%()content(5)delimiter(\))>operator(\))

 operator(})operator(.)ident(call)

 

@@ -911,7 +911,7 @@ ident(p) string<delimiter(%/)content(p)delimiter(/)>
 ident(p) string<delimiter(%Q[)content(<LI>)delimiter(])>

 ident(i)operator(=)integer(99)

 ident(p) string<delimiter(%Q[)content(<LI><A HREF=")inline<delimiter(#{)ident(i)operator([)integer(3)operator(])delimiter(})>content(.html)char(\\#)inline<delimiter(#{)ident(i)operator([)integer(4)operator(])delimiter(})>content(">)inline<delimiter(#{)ident(i)operator([)integer(0)operator(])operator(+)ident(i)operator([)integer(1)operator(])operator(+)operator(()ident(i)operator([)integer(2)operator(])operator(\))delimiter(})>content(</A>)char(\\n)delimiter(])>

-ident(p)operator(()symbol(:side=)operator(>)symbol(:top)operator(\))

+ident(p)operator(()symbol(:side)operator(=)operator(>)symbol(:top)operator(\))

 ident(p) string<delimiter(%w[)content(a b c

      d e f)delimiter(])>

 ident(p) string<delimiter(%w[)content(a b c)content(\\n)content(

diff --git a/test/ruby/strange.in.rb b/test/ruby/strange.in.rb
index 6ff93ee..03fec4b 100644
--- a/test/ruby/strange.in.rb
+++ b/test/ruby/strange.in.rb
@@ -10,6 +10,15 @@ while x<10000
 g=%w{}

 x=0

 

+=begin

+class Hello

+  @hi   #class-instance-variable

+  self <<class

+    attr_accessor :hi

+  end

+end

+=end

+

 while x<100

  puts"#{g[x]}"

  x+=1

diff --git a/test/ruby/strange.out.raydebug b/test/ruby/strange.out.raydebug
index 8a9137c..baa4955 100644
--- a/test/ruby/strange.out.raydebug
+++ b/test/ruby/strange.out.raydebug
@@ -10,6 +10,15 @@ comment(#und die Ausgabe folgt)
 ident(g)operator(=)string<delimiter(%w{)delimiter(})>

 ident(x)operator(=)integer(0)

 

+comment(=begin

+class Hello

+  @hi   #class-instance-variable

+  self <<class

+    attr_accessor :hi

+  end

+end

+=end)

+

 reserved(while) ident(x)operator(<)integer(100)

  ident(puts)string<delimiter(")inline<delimiter(#{)ident(g)operator([)ident(x)operator(])delimiter(})>delimiter(")>

  ident(x)operator(+=)integer(1)

diff --git a/test/ruby/test-fitter.out.raydebug b/test/ruby/test-fitter.out.raydebug
index 87a9e0b..0c56759 100644
--- a/test/ruby/test-fitter.out.raydebug
+++ b/test/ruby/test-fitter.out.raydebug
@@ -43,4 +43,4 @@ harmlesslineharmlessline
 harmlessline

 EOL

 

-)
\ No newline at end of file
+)>
\ No newline at end of file