From 604256ba5bdd0e7d707201d119db7035f478234d Mon Sep 17 00:00:00 2001 From: Tanner Prynn Date: Mon, 22 Feb 2016 16:34:46 -0600 Subject: Add api, command line, etc. documentation for custom lexer/formatter loading --- doc/docs/lexerdevelopment.rst | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) (limited to 'doc/docs/lexerdevelopment.rst') diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst index 2c868440..8c9b7baa 100644 --- a/doc/docs/lexerdevelopment.rst +++ b/doc/docs/lexerdevelopment.rst @@ -88,8 +88,38 @@ one. Adding and testing a new lexer ============================== -To make Pygments aware of your new lexer, you have to perform the following -steps: +The easiest way to use a new lexer is to use Pygments' support for loading +the lexer from a file relative to your current directory. + +First, change the name of your lexer class to CustomLexer: + +.. code-block:: python + + from pygments.lexer import RegexLexer + from pygments.token import * + + class CustomLexer(RegexLexer): + """All your lexer code goes here!""" + +Then you can load the lexer from the command line with the additional +flag ``--load-from-file``: + +.. code-block:: console + + $ pygmentize -l your_lexer_file.py --load-from-file + +Or, using the Python API: + +.. code-block:: python + + your_lexer = load_lexer_from_file(filename, **options) + +When loading custom lexers and formatters, be extremely careful to use only +trusted files; Pygments will perform the equivalent of ``eval`` on them. + +For more complicated tasks, you should add your lexer to Pygments. If you +want to make Pygments aware of your new lexer, you have to perform the +following steps: First, change to the current directory containing the Pygments source code: -- cgit v1.2.1 From 4a49415561c00e2d235d266ee81cb026982f8e20 Mon Sep 17 00:00:00 2001 From: Tanner Prynn Date: Wed, 24 Feb 2016 17:46:32 -0600 Subject: Update pull request per comments by birkenfeld. Add optional function parameter for the class name to instantiate, and update cli to support this. Move error handling to within the loading functions; they now only raise ClassNotFound. Modify doc with these updates and the version number. Test case clean up and additions. --- doc/docs/lexerdevelopment.rst | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) (limited to 'doc/docs/lexerdevelopment.rst') diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst index 8c9b7baa..bba81d24 100644 --- a/doc/docs/lexerdevelopment.rst +++ b/doc/docs/lexerdevelopment.rst @@ -102,18 +102,28 @@ First, change the name of your lexer class to CustomLexer: """All your lexer code goes here!""" Then you can load the lexer from the command line with the additional -flag ``--load-from-file``: +flag ``-x``: .. code-block:: console - $ pygmentize -l your_lexer_file.py --load-from-file + $ pygmentize -l your_lexer_file.py -x + +To specify a class name other than CustomLexer, append it with a colon: + +.. code-block:: console + + $ pygmentize -l your_lexer.py:SomeLexer -x Or, using the Python API: .. code-block:: python + # For a lexer named CustomLexer your_lexer = load_lexer_from_file(filename, **options) + # For a lexer named MyNewLexer + your_named_lexer = load_lexer_from_file(filename, "MyNewLexer", **options) + When loading custom lexers and formatters, be extremely careful to use only trusted files; Pygments will perform the equivalent of ``eval`` on them. -- cgit v1.2.1 From c8973a6f5a795a552c9630af2ef8aede1760fd1e Mon Sep 17 00:00:00 2001 From: Tanner Prynn Date: Mon, 29 Feb 2016 12:03:32 -0600 Subject: remove leftover documentation that I missed in the previous merge --- doc/docs/lexerdevelopment.rst | 2 -- 1 file changed, 2 deletions(-) (limited to 'doc/docs/lexerdevelopment.rst') diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst index cf51e7b1..c55c98a9 100644 --- a/doc/docs/lexerdevelopment.rst +++ b/doc/docs/lexerdevelopment.rst @@ -146,13 +146,11 @@ Select a matching module under ``pygments/lexers``, or create a new module for your lexer class. Next, make sure the lexer is known from outside of the module. All modules in -the ``pygments.lexers`` specify ``__all__``. For example, ``esoteric.py`` sets:: the ``pygments.lexers`` package specify ``__all__``. For example, ``esoteric.py`` sets:: __all__ = ['BrainfuckLexer', 'BefungeLexer', ...] -Simply add the name of your lexer class to this list. Add the name of your lexer class to this list (or create the list if your lexer is the only class in the module). -- cgit v1.2.1 From 7c0f54f70958fac9c3bf181b783e74abadecea39 Mon Sep 17 00:00:00 2001 From: cocoatomo Date: Tue, 14 Jun 2016 01:33:42 +0900 Subject: Insert a missing argument "self" --- doc/docs/lexerdevelopment.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/docs/lexerdevelopment.rst') diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst index fd6e76b9..9109180d 100644 --- a/doc/docs/lexerdevelopment.rst +++ b/doc/docs/lexerdevelopment.rst @@ -361,7 +361,7 @@ There are a few more things you can do with states: tokens = {...} def get_tokens_unprocessed(self, text, stack=('root', 'otherstate')): - for item in RegexLexer.get_tokens_unprocessed(text, stack): + for item in RegexLexer.get_tokens_unprocessed(self, text, stack): yield item Some lexers like the `PhpLexer` use this to make the leading ``