New: *Simple Diff Scanner* (closes #22).

* Highlights unified diffs, especially like the ones svn diff outputs.
* Changes to make highlighting of whole lines were necessary.
* I added two example files.

More changes:
* Added token classes :head, :delete, :insert, and :change along with styles.
* Added two new special token types: :begin_line and :end_line. They mark token groups that explicitly span whole lines and should be highlighted as such.
* The HTML encoder converts these new tokens to DIVs. May need more work.
* The Debug Encoder uses square brackets for line tokens.
* Some cleanups.

1 files changed, 14 insertions, 10 deletions

diff --git a/lib/coderay/tokens.rb b/lib/coderay/tokens.rb
index 7d82afb..50d324c 100644
--- a/lib/coderay/tokens.rb
+++ b/lib/coderay/tokens.rb
@@ -200,25 +200,29 @@ module CodeRay
     #
     # TODO: Test this!
     def fix
+      tokens = self.class.new
       # Check token nesting using a stack of kinds.
       opened = []
-      for token, kind in self
-        if token == :open
-          opened.push kind
-        elsif token == :close
+      for type, kind in self
+        case type
+        when :open
+          opened.push [:close, kind]
+        when :begin_line
+          opened.push [:end_line, kind]
+        when :close, :end_line
           expected = opened.pop
-          if kind != expected
+          if [type, kind] != expected
             # Unexpected :close; decide what to do based on the kind:
-            # - token was opened earlier: also close tokens in between
-            # - token was never opened: delete the :close (skip with next)
+            # - token was never opened: delete the :close (just skip it)
             next unless opened.rindex expected
-            tokens << [:close, kind] until (kind = opened.pop) == expected
+            # - token was opened earlier: also close tokens in between
+            tokens << token until (token = opened.pop) == expected
           end
         end
-        tokens << [token, kind]
+        tokens << [type, kind]
       end
       # Close remaining opened tokens
-      tokens << [:close, kind] while kind = opened.pop
+      tokens << token while token = opened.pop
       tokens
     end