Done:

General:
- Declared version 0.7.6.
- Moved WordList, CaseIgnoringWordList, Plugin, PluginHost and FileType
  into CodeRay namespace. CodeRay should be "clean" now, except for the
  String#to_unix helper function.
- Fixed a bit of documentation.
- CodeRay binary: Prepare for streaming switch.

Scanners:
- Added code= alias for string=.
- Added streaming? method: Is this Scanner in streaming mode?
- Enhanced error info a bit.
- Ruby scanner:
  - Highlights Regexp heredocs now. They may be added to Ruby 1.9.
  - Speedups with better support for Ruby 1.9.
  - Change in whitespace handling (faster and cleaner now.)
  - Speed up some operator recognition (saving two string comparisons).
- Declared C and Plaintext Scanners as Streamable.

Tokens:
- Changed Text/Block token recognition (#is_a? ::String for Ruby 1.9 support).
- New method: Tokens#text yields the code string.
- text_size fixed.
- Token kind shortcuts (like r for reserved) are now defined in
  token_classes.rb (instead of encoders/html/classes.rb).

Encoders:
- Debug Scanner added.
- Base encoder class adds to @out when encoding (if @out is set).
- A little Tokens scanner speedup.
- Text encoder uses text_token.
- Statistic encoder counts block tokens.
- Smaller changes in XML and HTML encoders.

Styles:
- cYcnus style defines a debug class now.

Duo:
- scanner and encoder are now methods. Scanner and Encoder are created (and
  cached) when needed, not earlier.
- Documented.

Tests:
- Disabled encoder and scanner list check (breaks too often).
- Added identity test, which checks if tokens#text matches the input.
- Added nocolor switch.

Developer tools:
- Benchmark uses Encoder#file_extension for output now.
- Rakefile: Support for 19, 18, yarv and ruby switches for easy comparing
  different Ruby versions.
- Statistic: Demos are no longer tests.

1 files changed, 14 insertions, 3 deletions

diff --git a/lib/coderay/tokens.rb b/lib/coderay/tokens.rb
index b0ce70e..d05177a 100644
--- a/lib/coderay/tokens.rb
+++ b/lib/coderay/tokens.rb
@@ -115,7 +115,7 @@ module CodeRay
     #   tokens.each_text_token { |text, kind| text.replace html_escape(text) }
     def each_text_token
       each do |text, kind|
-        next unless text.respond_to? :to_str
+        next unless text.is_a? ::String
         yield text, kind
       end
     end
@@ -252,7 +252,7 @@ module CodeRay
     #
     # You can configure the level of compression,
     # but the default value 7 should be what you want
-    # in most cases as it is a good comprimise between
+    # in most cases as it is a good compromise between
     # speed and compression rate.
     #
     # See GZip module.
@@ -267,7 +267,14 @@ module CodeRay
     # Should be equal to the input size before
     # scanning.
     def text_size
-      map { |t, k| t }.join.size
+      inject(0) { |size, (t, k)| t.is_a?(::String) ? size : size + t.size }
+    end
+
+    # The total size of the tokens.
+    # Should be equal to the input size before
+    # scanning.
+    def text
+      map { |t, k| t if t.is_a? ::String }.join
     end
 
     # Include this module to give an object an #undump
@@ -365,4 +372,8 @@ module CodeRay
 
   end
 
+  
+  # Token name abbreviations
+  require 'coderay/token_classes'
+
 end