Done:

General:
- Declared version 0.7.6.
- Moved WordList, CaseIgnoringWordList, Plugin, PluginHost and FileType
  into CodeRay namespace. CodeRay should be "clean" now, except for the
  String#to_unix helper function.
- Fixed a bit of documentation.
- CodeRay binary: Prepare for streaming switch.

Scanners:
- Added code= alias for string=.
- Added streaming? method: Is this Scanner in streaming mode?
- Enhanced error info a bit.
- Ruby scanner:
  - Highlights Regexp heredocs now. They may be added to Ruby 1.9.
  - Speedups with better support for Ruby 1.9.
  - Change in whitespace handling (faster and cleaner now.)
  - Speed up some operator recognition (saving two string comparisons).
- Declared C and Plaintext Scanners as Streamable.

Tokens:
- Changed Text/Block token recognition (#is_a? ::String for Ruby 1.9 support).
- New method: Tokens#text yields the code string.
- text_size fixed.
- Token kind shortcuts (like r for reserved) are now defined in
  token_classes.rb (instead of encoders/html/classes.rb).

Encoders:
- Debug Scanner added.
- Base encoder class adds to @out when encoding (if @out is set).
- A little Tokens scanner speedup.
- Text encoder uses text_token.
- Statistic encoder counts block tokens.
- Smaller changes in XML and HTML encoders.

Styles:
- cYcnus style defines a debug class now.

Duo:
- scanner and encoder are now methods. Scanner and Encoder are created (and
  cached) when needed, not earlier.
- Documented.

Tests:
- Disabled encoder and scanner list check (breaks too often).
- Added identity test, which checks if tokens#text matches the input.
- Added nocolor switch.

Developer tools:
- Benchmark uses Encoder#file_extension for output now.
- Rakefile: Support for 19, 18, yarv and ruby switches for easy comparing
  different Ruby versions.
- Statistic: Demos are no longer tests.

8 files changed, 29 insertions, 118 deletions

diff --git a/lib/coderay/encoders/_map.rb b/lib/coderay/encoders/_map.rb
index fdd8ae4..8e9732b 100644
--- a/lib/coderay/encoders/_map.rb
+++ b/lib/coderay/encoders/_map.rb
@@ -2,7 +2,8 @@ module CodeRay
 module Encoders
 
   map :stats => :statistic,
-    :plain => :text
+    :plain => :text,
+    :tex => :latex
 
 end
 end
diff --git a/lib/coderay/encoders/debug.rb b/lib/coderay/encoders/debug.rb
index eb9eaa4..21d4710 100644
--- a/lib/coderay/encoders/debug.rb
+++ b/lib/coderay/encoders/debug.rb
@@ -19,17 +19,12 @@ module Encoders
 
   protected
     def text_token text, kind
-      @out <<
-        if kind == :space
-          text
-        else
-          text = text.gsub(/[)\\]/, '\\\\\0')
-          "#{kind}(#{text})"
-        end
-    end
-
-    def block_token action, kind
-      @out << super
+      if kind == :space
+        text
+      else
+        text = text.gsub(/[)\\]/, '\\\\\0')
+        "#{kind}(#{text})"
+      end
     end
 
     def open_token kind
diff --git a/lib/coderay/encoders/html.rb b/lib/coderay/encoders/html.rb
index 3ee677a..32e35f8 100644
--- a/lib/coderay/encoders/html.rb
+++ b/lib/coderay/encoders/html.rb
@@ -86,7 +86,7 @@ module Encoders
       :hint => false,
     }
 
-    helper :classes, :output, :css
+    helper :output, :css
 
     attr_reader :css
 
@@ -119,15 +119,14 @@ module Encoders
         end
     }
 
-    TRANSPARENT_TOKEN_KINDS = Set[
+    TRANSPARENT_TOKEN_KINDS = [
       :delimiter, :modifier, :content, :escape, :inline_delimiter,
-    ]
+    ].to_set
 
     # Generate a hint about the given +classes+ in a +hint+ style.
     #
     # +hint+ may be :info, :info_long or :debug.
     def self.token_path_to_hint hint, classes
-      return '' unless hint
       title =
         case hint
         when :info
@@ -159,29 +158,28 @@ module Encoders
 
       when :class
         @css_style = Hash.new do |h, k|
-          if k.is_a? Array
-            type = k.first
-          else
-            type = k
-          end
-          c = ClassOfKind[type]
+          c = Tokens::ClassOfKind[k.first]
           if c == :NO_HIGHLIGHT and not hint
-            h[k] = false
+            h[k.dup] = false
           else
-            title = HTML.token_path_to_hint hint, (k[1..-1] << k.first)
-            h[k] = '<span%s class="%s">' % [title, c]
+            title = if hint
+              HTML.token_path_to_hint(hint, k[1..-1] << k.first)
+            else
+              ''
+            end
+            h[k.dup] = '<span%s class="%s">' % [title, c]
           end
         end
 
       when :style
         @css_style = Hash.new do |h, k|
-          if k.is_a? Array
+          if k.is_a? ::Array
             styles = k.dup
           else
             styles = [k]
           end
           type = styles.first
-          classes = styles.map { |c| ClassOfKind[c] }
+          classes = styles.map { |c| Tokens::ClassOfKind[c] }
           if classes.first == :NO_HIGHLIGHT and not hint
             h[k] = false
           else
diff --git a/lib/coderay/encoders/html/classes.rb b/lib/coderay/encoders/html/classes.rb
deleted file mode 100644
index 0bac742..0000000
--- a/lib/coderay/encoders/html/classes.rb
+++ /dev/null
@@ -1,77 +0,0 @@
-module CodeRay
-module Encoders
-
-  class HTML
-
-    ClassOfKind = Hash.new do |h, k|
-      h[k] = k.to_s
-    end
-    ClassOfKind.update with = {
-      :attribute_name => 'an',
-      :attribute_name_fat => 'af',
-      :attribute_value => 'av',
-      :attribute_value_fat => 'aw',
-      :bin => 'bi',
-      :char => 'ch',
-      :class => 'cl',
-      :class_variable => 'cv',
-      :color => 'cr',
-      :comment => 'c',
-      :constant => 'co',
-      :content => 'k',
-      :definition => 'df',
-      :delimiter => 'dl',
-      :directive => 'di',
-      :doc => 'do',
-      :doc_string => 'ds',
-      :entity => 'en',
-      :error => 'er',
-      :escape => 'e',
-      :exception => 'ex',
-      :float => 'fl',
-      :function => 'fu',
-      :global_variable => 'gv',
-      :hex => 'hx',
-      :include => 'ic',
-      :inline => 'il',
-      :inline_delimiter => 'idl',
-      :instance_variable => 'iv',
-      :integer => 'i',
-      :interpreted => 'in',
-      :label => 'la',
-      :local_variable => 'lv',
-      :modifier => 'mod',
-      :oct => 'oc',
-      :operator_name => 'on',
-      :pre_constant => 'pc',
-      :pre_type => 'pt',
-      :predefined => 'pd',
-      :preprocessor => 'pp',
-      :regexp => 'rx',
-      :reserved => 'r',
-      :shell => 'sh',
-      :string => 's',
-      :symbol => 'sy',
-      :tag => 'ta',
-      :tag_fat => 'tf',
-      :tag_special => 'ts',
-      :type => 'ty',
-      :variable => 'v',
-      :xml_text => 'xt',
-
-      :ident => :NO_HIGHLIGHT, # 'id'
-      #:operator => 'op',
-      :operator => :NO_HIGHLIGHT,  # 'op'
-      :space => :NO_HIGHLIGHT,  # 'sp'
-      :plain => :NO_HIGHLIGHT,
-    }
-    ClassOfKind[:procedure] = ClassOfKind[:method] = ClassOfKind[:function]
-    ClassOfKind[:open] = ClassOfKind[:close] = ClassOfKind[:delimiter]
-    ClassOfKind[:nesting_delimiter] = ClassOfKind[:delimiter]
-    ClassOfKind[:escape] = ClassOfKind[:delimiter]
-    #ClassOfKind.default = ClassOfKind[:error] or raise 'no class found for :error!'
-
-  end
-
-end
-end
diff --git a/lib/coderay/encoders/statistic.rb b/lib/coderay/encoders/statistic.rb
index e2a0460..6d0c646 100644
--- a/lib/coderay/encoders/statistic.rb
+++ b/lib/coderay/encoders/statistic.rb
@@ -28,17 +28,13 @@ module Encoders
       @type_stats[kind].count += 1
       @type_stats[kind].size += text.size
       @type_stats['TOTAL'].size += text.size
+      @type_stats['TOTAL'].count += 1
     end
 
     # TODO Hierarchy handling
     def block_token action, kind
-      #@content_type = kind
-      @type_stats['open/close'].count += 1
-    end
-
-    def token text, kind
-      super
       @type_stats['TOTAL'].count += 1
+      @type_stats['open/close'].count += 1
     end
 
     STATS = <<-STATS
diff --git a/lib/coderay/encoders/text.rb b/lib/coderay/encoders/text.rb
index 17256c6..7493280 100644
--- a/lib/coderay/encoders/text.rb
+++ b/lib/coderay/encoders/text.rb
@@ -18,9 +18,8 @@ module Encoders
       @sep = options[:separator]
     end
 
-    def token text, kind
-      return unless text.respond_to? :to_str
-      @out << text + @sep
+    def text_token text, kind
+      text + @sep
     end
 
     def finish options
diff --git a/lib/coderay/encoders/tokens.rb b/lib/coderay/encoders/tokens.rb
index 2428589..27c7f6d 100644
--- a/lib/coderay/encoders/tokens.rb
+++ b/lib/coderay/encoders/tokens.rb
@@ -33,9 +33,9 @@ module Encoders
 
     FILE_EXTENSION = 'tok'
 
-    protected
-    def token *args
-      @out << CodeRay::Tokens.write_token(*args)
+  protected
+    def token text, kind
+      @out << CodeRay::Tokens.write_token(text, kind)
     end
 
   end
diff --git a/lib/coderay/encoders/xml.rb b/lib/coderay/encoders/xml.rb
index 09e4549..dffa98c 100644
--- a/lib/coderay/encoders/xml.rb
+++ b/lib/coderay/encoders/xml.rb
@@ -22,7 +22,6 @@ module Encoders
   protected
 
     def setup options
-      @out = ''
       @doc = REXML::Document.new
       @doc << REXML::XMLDecl.new
       @tab_width = options[:tab_width]
@@ -33,7 +32,7 @@ module Encoders
       @doc.write @out, options[:pretty], options[:transitive], true
       @out
     end
-
+    
     def text_token text, kind
       if kind == :space
         token = @node