Done:

General:
- Declared version 0.7.6.
- Moved WordList, CaseIgnoringWordList, Plugin, PluginHost and FileType
  into CodeRay namespace. CodeRay should be "clean" now, except for the
  String#to_unix helper function.
- Fixed a bit of documentation.
- CodeRay binary: Prepare for streaming switch.

Scanners:
- Added code= alias for string=.
- Added streaming? method: Is this Scanner in streaming mode?
- Enhanced error info a bit.
- Ruby scanner:
  - Highlights Regexp heredocs now. They may be added to Ruby 1.9.
  - Speedups with better support for Ruby 1.9.
  - Change in whitespace handling (faster and cleaner now.)
  - Speed up some operator recognition (saving two string comparisons).
- Declared C and Plaintext Scanners as Streamable.

Tokens:
- Changed Text/Block token recognition (#is_a? ::String for Ruby 1.9 support).
- New method: Tokens#text yields the code string.
- text_size fixed.
- Token kind shortcuts (like r for reserved) are now defined in
  token_classes.rb (instead of encoders/html/classes.rb).

Encoders:
- Debug Scanner added.
- Base encoder class adds to @out when encoding (if @out is set).
- A little Tokens scanner speedup.
- Text encoder uses text_token.
- Statistic encoder counts block tokens.
- Smaller changes in XML and HTML encoders.

Styles:
- cYcnus style defines a debug class now.

Duo:
- scanner and encoder are now methods. Scanner and Encoder are created (and
  cached) when needed, not earlier.
- Documented.

Tests:
- Disabled encoder and scanner list check (breaks too often).
- Added identity test, which checks if tokens#text matches the input.
- Added nocolor switch.

Developer tools:
- Benchmark uses Encoder#file_extension for output now.
- Rakefile: Support for 19, 18, yarv and ruby switches for easy comparing
  different Ruby versions.
- Statistic: Demos are no longer tests.

4 files changed, 36 insertions, 32 deletions

diff --git a/lib/coderay/scanners/c.rb b/lib/coderay/scanners/c.rb
index be113d0..2b63a81 100644
--- a/lib/coderay/scanners/c.rb
+++ b/lib/coderay/scanners/c.rb
@@ -4,6 +4,8 @@ module Scanners
   class C < Scanner
 
     register_for :c
+    
+    include Streamable
 
     RESERVED_WORDS = [
       'asm', 'break', 'case', 'continue', 'default', 'do', 'else',
diff --git a/lib/coderay/scanners/plaintext.rb b/lib/coderay/scanners/plaintext.rb
index 1cd7a8a..7a08c3a 100644
--- a/lib/coderay/scanners/plaintext.rb
+++ b/lib/coderay/scanners/plaintext.rb
@@ -4,6 +4,8 @@ module Scanners
   class Plaintext < Scanner
 
     register_for :plaintext, :plain
+    
+    include Streamable
 
     def scan_tokens tokens, options
       text = (scan_until(/\z/) || '')
diff --git a/lib/coderay/scanners/ruby.rb b/lib/coderay/scanners/ruby.rb
index def5960..b373a2b 100644
--- a/lib/coderay/scanners/ruby.rb
+++ b/lib/coderay/scanners/ruby.rb
@@ -90,15 +90,15 @@ module Scanners
             end
 
           when '#'
-            case peek(1)[0]
-            when ?{
+            case peek(1)
+            when '{'
               inline_block_stack << [state, depth, heredocs]
               value_expected = true
               state = :initial
               depth = 1
               tokens << [:open, :inline]
               tokens << [match + getch, :inline_delimiter]
-            when ?$, ?@
+            when '$', '@'
               tokens << [match, :escape]
               last_state = state  # scan one token as normal code, then return here
               state = :initial
@@ -121,36 +121,36 @@ module Scanners
 # }}}
         else
 # {{{
-          if match = scan(/ [ \t\f]+ | \\? \n | \# .* /x) or
-            ( bol? and match = scan(/#{patterns::RUBYDOC_OR_DATA}/o) )
-            case m = match[0]
-            when ?\s, ?\t, ?\f
-              match << scan(/\s*/) unless eos? or heredocs
-              kind = :space
-            when ?\n, ?\\
-              kind = :space
-              if m == ?\n
-                value_expected = true  # FIXME not quite true
-                state = :initial if state == :undef_comma_expected
-              end
-              if heredocs
-                unscan  # heredoc scanning needs \n at start
-                state = heredocs.shift
-                tokens << [:open, state.type]
-                heredocs = nil if heredocs.empty?
-                next
-              else
-                match << scan(/\s*/) unless eos?
-              end
-            when ?#, ?=, ?_
-              kind = :comment
-              value_expected = true
+          if match = scan(/[ \t\f]+/)
+            kind = :space
+            match << scan(/\s*/) unless eos? or heredocs
+            tokens << [match, kind]
+            next
+            
+          elsif match = scan(/\\?\n/)
+            kind = :space
+            if match == "\n"
+              value_expected = true  # FIXME not quite true
+              state = :initial if state == :undef_comma_expected
+            end
+            if heredocs
+              unscan  # heredoc scanning needs \n at start
+              state = heredocs.shift
+              tokens << [:open, state.type]
+              heredocs = nil if heredocs.empty?
+              next
             else
-              raise_inspect 'else-case _ reached, because case %p was
-                not handled' % [matched[0].chr], tokens
+              match << scan(/\s*/) unless eos?
             end
             tokens << [match, kind]
             next
+          
+          elsif match = scan(/\#.*/) or
+            ( bol? and match = scan(/#{patterns::RUBYDOC_OR_DATA}/o) )
+              kind = :comment
+              value_expected = true
+              tokens << [match, kind]
+              next
 
           elsif state == :initial
 
@@ -175,11 +175,11 @@ module Scanners
               value_expected = :set if check(/#{patterns::VALUE_FOLLOWS}/o)
 
             # OPERATORS #
-            elsif not last_token_dot and match = scan(/ ==?=? | \.\.?\.? | [\(\)\[\]\{\}] | :: | , /x)
+            elsif not last_token_dot and match = scan(/ \.\.\.? | (?:\.|::)() | [,\(\)\[\]\{\}] | ==?=? /x)
               if match !~ / [.\)\]\}] /x or match =~ /\.\.\.?/
                 value_expected = :set
               end
-              last_token_dot = :set if match == '.' or match == '::'
+              last_token_dot = :set if self[1]
               kind = :operator
               unless inline_block_stack.empty?
                 case match
diff --git a/lib/coderay/scanners/ruby/patterns.rb b/lib/coderay/scanners/ruby/patterns.rb
index 51cdb95..39962ec 100644
--- a/lib/coderay/scanners/ruby/patterns.rb
+++ b/lib/coderay/scanners/ruby/patterns.rb
@@ -111,7 +111,7 @@ module Scanners
       (?:
         ( [A-Za-z_0-9]+ )  # $2 = delim
       |
-        ( ["'`] )          # $3 = quote, type
+        ( ["'`\/] )        # $3 = quote, type
         ( [^\n]*? ) \3     # $4 = delim
       )
     /mx