From 715d40c86c01160513e2bd21f6f8807f343938da Mon Sep 17 00:00:00 2001 From: Tanner Prynn Date: Mon, 22 Feb 2016 13:37:30 -0600 Subject: Add additional command line option to prevent users from using eval() on untrusted files Finish custom-formatter loading and fill in some docstrings Add load_?_from_file functions to API documentation pep8 compliance --- doc/docs/api.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) (limited to 'doc/docs') diff --git a/doc/docs/api.rst b/doc/docs/api.rst index dd831bd1..1b6ca668 100644 --- a/doc/docs/api.rst +++ b/doc/docs/api.rst @@ -62,6 +62,18 @@ Functions from :mod:`pygments.lexers`: Will raise :exc:`pygments.util.ClassNotFound` if not lexer for that mimetype is found. +.. function:: load_lexer_from_file(filename, **options) + + Return a `Lexer` subclass instance loaded from the provided file, relative + to the current directory. The file is expected to contain a CustomLexer class + which matches Pygments' lexer definitions. Users should be very careful with + the input, because this method is equivalent to running eval on the input file. + The lexer is given the `options` at its instantiation. + + :exc:`IOError` is raised if the file is not found or unreadable + :exc:`ImportError` is raised if the file doesn't have a CustomLexer class + :exc:`Exception` for any other error raised when evaluating filename + .. function:: guess_lexer(text, **options) Return a `Lexer` subclass instance that's guessed from the text in @@ -125,6 +137,18 @@ Functions from :mod:`pygments.formatters`: Will raise :exc:`pygments.util.ClassNotFound` if no formatter for that filename is found. +.. function:: load_formatter_from_file(filename, **options) + + Return a `Formatter` subclass instance loaded from the provided file, relative + to the current directory. The file is expected to contain a CustomFormatter class + which matches Pygments' formatter definitions. Users should be very careful with + the input, because this method is equivalent to running eval on the input file. + The formatter is given the `options` at its instantiation. + + :exc:`IOError` is raised if the file is not found or unreadable + :exc:`ImportError` is raised if the file doesn't have a CustomFormatter class + :exc:`Exception` for any other error raised when evaluating filename + .. module:: pygments.styles -- cgit v1.2.1 From 604256ba5bdd0e7d707201d119db7035f478234d Mon Sep 17 00:00:00 2001 From: Tanner Prynn Date: Mon, 22 Feb 2016 16:34:46 -0600 Subject: Add api, command line, etc. documentation for custom lexer/formatter loading --- doc/docs/api.rst | 11 +++++++++-- doc/docs/cmdline.rst | 13 +++++++++++++ doc/docs/lexerdevelopment.rst | 34 ++++++++++++++++++++++++++++++++-- 3 files changed, 54 insertions(+), 4 deletions(-) (limited to 'doc/docs') diff --git a/doc/docs/api.rst b/doc/docs/api.rst index 1b6ca668..ea158d65 100644 --- a/doc/docs/api.rst +++ b/doc/docs/api.rst @@ -71,8 +71,12 @@ Functions from :mod:`pygments.lexers`: The lexer is given the `options` at its instantiation. :exc:`IOError` is raised if the file is not found or unreadable + :exc:`ImportError` is raised if the file doesn't have a CustomLexer class - :exc:`Exception` for any other error raised when evaluating filename + + Additionally, arbitrary exceptions could occur when evaluating the file + + .. versionadded:: ? .. function:: guess_lexer(text, **options) @@ -146,9 +150,12 @@ Functions from :mod:`pygments.formatters`: The formatter is given the `options` at its instantiation. :exc:`IOError` is raised if the file is not found or unreadable + :exc:`ImportError` is raised if the file doesn't have a CustomFormatter class - :exc:`Exception` for any other error raised when evaluating filename + Additionally, arbitrary exceptions could occur when evaluating the file + + .. versionadded:: ? .. module:: pygments.styles diff --git a/doc/docs/cmdline.rst b/doc/docs/cmdline.rst index 165af969..8fcd3c8c 100644 --- a/doc/docs/cmdline.rst +++ b/doc/docs/cmdline.rst @@ -99,6 +99,19 @@ The ``-N`` option guesses a lexer name for a given filename, so that :: will print out ``python``. It won't highlight anything yet. If no specific lexer is known for that filename, ``text`` is printed. +Custom Lexers and Formatters +---------------------------- + +.. versionadded:: ? + +The ``--load-from-file`` flag enables custom lexers and formatters to be loaded +from files relative to the current directory. Create a file with a class named +CustomLexer or CustomFormatter, then specify it on the command line:: + + $ pygmentize -l your_lexer.py -f your_formatter.py --load-from-file + +For more information, see :doc:`the Pygments documentation on Lexer development +`. Getting help ------------ diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst index 2c868440..8c9b7baa 100644 --- a/doc/docs/lexerdevelopment.rst +++ b/doc/docs/lexerdevelopment.rst @@ -88,8 +88,38 @@ one. Adding and testing a new lexer ============================== -To make Pygments aware of your new lexer, you have to perform the following -steps: +The easiest way to use a new lexer is to use Pygments' support for loading +the lexer from a file relative to your current directory. + +First, change the name of your lexer class to CustomLexer: + +.. code-block:: python + + from pygments.lexer import RegexLexer + from pygments.token import * + + class CustomLexer(RegexLexer): + """All your lexer code goes here!""" + +Then you can load the lexer from the command line with the additional +flag ``--load-from-file``: + +.. code-block:: console + + $ pygmentize -l your_lexer_file.py --load-from-file + +Or, using the Python API: + +.. code-block:: python + + your_lexer = load_lexer_from_file(filename, **options) + +When loading custom lexers and formatters, be extremely careful to use only +trusted files; Pygments will perform the equivalent of ``eval`` on them. + +For more complicated tasks, you should add your lexer to Pygments. If you +want to make Pygments aware of your new lexer, you have to perform the +following steps: First, change to the current directory containing the Pygments source code: -- cgit v1.2.1 From 4a49415561c00e2d235d266ee81cb026982f8e20 Mon Sep 17 00:00:00 2001 From: Tanner Prynn Date: Wed, 24 Feb 2016 17:46:32 -0600 Subject: Update pull request per comments by birkenfeld. Add optional function parameter for the class name to instantiate, and update cli to support this. Move error handling to within the loading functions; they now only raise ClassNotFound. Modify doc with these updates and the version number. Test case clean up and additions. --- doc/docs/api.rst | 32 ++++++++++++-------------------- doc/docs/cmdline.rst | 10 +++++++--- doc/docs/lexerdevelopment.rst | 14 ++++++++++++-- 3 files changed, 31 insertions(+), 25 deletions(-) (limited to 'doc/docs') diff --git a/doc/docs/api.rst b/doc/docs/api.rst index ea158d65..a6b242dd 100644 --- a/doc/docs/api.rst +++ b/doc/docs/api.rst @@ -62,21 +62,17 @@ Functions from :mod:`pygments.lexers`: Will raise :exc:`pygments.util.ClassNotFound` if not lexer for that mimetype is found. -.. function:: load_lexer_from_file(filename, **options) +.. function:: load_lexer_from_file(filename, lexername="CustomLexer", **options) Return a `Lexer` subclass instance loaded from the provided file, relative - to the current directory. The file is expected to contain a CustomLexer class - which matches Pygments' lexer definitions. Users should be very careful with + to the current directory. The file is expected to contain a Lexer class + named `lexername` (by default, CustomLexer). Users should be very careful with the input, because this method is equivalent to running eval on the input file. The lexer is given the `options` at its instantiation. - :exc:`IOError` is raised if the file is not found or unreadable + :exc:`ClassNotFound` is raised if there are any errors loading the Lexer - :exc:`ImportError` is raised if the file doesn't have a CustomLexer class - - Additionally, arbitrary exceptions could occur when evaluating the file - - .. versionadded:: ? + .. versionadded:: 2.2 .. function:: guess_lexer(text, **options) @@ -141,21 +137,17 @@ Functions from :mod:`pygments.formatters`: Will raise :exc:`pygments.util.ClassNotFound` if no formatter for that filename is found. -.. function:: load_formatter_from_file(filename, **options) +.. function:: load_formatter_from_file(filename, formattername="CustomFormatter", **options) Return a `Formatter` subclass instance loaded from the provided file, relative - to the current directory. The file is expected to contain a CustomFormatter class - which matches Pygments' formatter definitions. Users should be very careful with - the input, because this method is equivalent to running eval on the input file. - The formatter is given the `options` at its instantiation. + to the current directory. The file is expected to contain a Formatter class + named ``formattername`` (by default, CustomFormatter). Users should be very + careful with the input, because this method is equivalent to running eval + on the input file. The formatter is given the `options` at its instantiation. - :exc:`IOError` is raised if the file is not found or unreadable + :exc:`ClassNotFound` is raised if there are any errors loading the Formatter - :exc:`ImportError` is raised if the file doesn't have a CustomFormatter class - - Additionally, arbitrary exceptions could occur when evaluating the file - - .. versionadded:: ? + .. versionadded:: 2.2 .. module:: pygments.styles diff --git a/doc/docs/cmdline.rst b/doc/docs/cmdline.rst index 8fcd3c8c..e4f94ea5 100644 --- a/doc/docs/cmdline.rst +++ b/doc/docs/cmdline.rst @@ -102,13 +102,17 @@ lexer is known for that filename, ``text`` is printed. Custom Lexers and Formatters ---------------------------- -.. versionadded:: ? +.. versionadded:: 2.2 -The ``--load-from-file`` flag enables custom lexers and formatters to be loaded +The ``-x`` flag enables custom lexers and formatters to be loaded from files relative to the current directory. Create a file with a class named CustomLexer or CustomFormatter, then specify it on the command line:: - $ pygmentize -l your_lexer.py -f your_formatter.py --load-from-file + $ pygmentize -l your_lexer.py -f your_formatter.py -x + +You can also specify the name of your class with a colon:: + + $ pygmentize -l your_lexer.py:SomeLexer -x For more information, see :doc:`the Pygments documentation on Lexer development `. diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst index 8c9b7baa..bba81d24 100644 --- a/doc/docs/lexerdevelopment.rst +++ b/doc/docs/lexerdevelopment.rst @@ -102,18 +102,28 @@ First, change the name of your lexer class to CustomLexer: """All your lexer code goes here!""" Then you can load the lexer from the command line with the additional -flag ``--load-from-file``: +flag ``-x``: .. code-block:: console - $ pygmentize -l your_lexer_file.py --load-from-file + $ pygmentize -l your_lexer_file.py -x + +To specify a class name other than CustomLexer, append it with a colon: + +.. code-block:: console + + $ pygmentize -l your_lexer.py:SomeLexer -x Or, using the Python API: .. code-block:: python + # For a lexer named CustomLexer your_lexer = load_lexer_from_file(filename, **options) + # For a lexer named MyNewLexer + your_named_lexer = load_lexer_from_file(filename, "MyNewLexer", **options) + When loading custom lexers and formatters, be extremely careful to use only trusted files; Pygments will perform the equivalent of ``eval`` on them. -- cgit v1.2.1 From c8973a6f5a795a552c9630af2ef8aede1760fd1e Mon Sep 17 00:00:00 2001 From: Tanner Prynn Date: Mon, 29 Feb 2016 12:03:32 -0600 Subject: remove leftover documentation that I missed in the previous merge --- doc/docs/lexerdevelopment.rst | 2 -- 1 file changed, 2 deletions(-) (limited to 'doc/docs') diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst index cf51e7b1..c55c98a9 100644 --- a/doc/docs/lexerdevelopment.rst +++ b/doc/docs/lexerdevelopment.rst @@ -146,13 +146,11 @@ Select a matching module under ``pygments/lexers``, or create a new module for your lexer class. Next, make sure the lexer is known from outside of the module. All modules in -the ``pygments.lexers`` specify ``__all__``. For example, ``esoteric.py`` sets:: the ``pygments.lexers`` package specify ``__all__``. For example, ``esoteric.py`` sets:: __all__ = ['BrainfuckLexer', 'BefungeLexer', ...] -Simply add the name of your lexer class to this list. Add the name of your lexer class to this list (or create the list if your lexer is the only class in the module). -- cgit v1.2.1 From 7c0f54f70958fac9c3bf181b783e74abadecea39 Mon Sep 17 00:00:00 2001 From: cocoatomo Date: Tue, 14 Jun 2016 01:33:42 +0900 Subject: Insert a missing argument "self" --- doc/docs/lexerdevelopment.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/docs') diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst index fd6e76b9..9109180d 100644 --- a/doc/docs/lexerdevelopment.rst +++ b/doc/docs/lexerdevelopment.rst @@ -361,7 +361,7 @@ There are a few more things you can do with states: tokens = {...} def get_tokens_unprocessed(self, text, stack=('root', 'otherstate')): - for item in RegexLexer.get_tokens_unprocessed(text, stack): + for item in RegexLexer.get_tokens_unprocessed(self, text, stack): yield item Some lexers like the `PhpLexer` use this to make the leading ``