Merge pull request #9822 from jakobandersen/intersphinx_role

Intersphinx role (2)

7 files changed, 337 insertions, 41 deletions

diff --git a/CHANGES b/CHANGES
index d2ddae71c..54b09e556 100644
--- a/CHANGES
+++ b/CHANGES
@@ -47,6 +47,8 @@ Features added
 * #9391: texinfo: improve variable in ``samp`` role
 * #9578: texinfo: Add :confval:`texinfo_cross_references` to disable cross
   references for readability with standalone readers
+* #9822 (and #9062), add new Intersphinx role :rst:role:`external` for explict
+  lookup in the external projects, without resolving to the local project.
 
 Bugs fixed
 ----------
diff --git a/doc/usage/extensions/intersphinx.rst b/doc/usage/extensions/intersphinx.rst
index a3e65bed6..2bcce68d0 100644
--- a/doc/usage/extensions/intersphinx.rst
+++ b/doc/usage/extensions/intersphinx.rst
@@ -8,20 +8,25 @@
 
 .. versionadded:: 0.5
 
-This extension can generate automatic links to the documentation of objects in
-other projects.
-
-Usage is simple: whenever Sphinx encounters a cross-reference that has no
-matching target in the current documentation set, it looks for targets in the
-documentation sets configured in :confval:`intersphinx_mapping`.  A reference
-like ``:py:class:`zipfile.ZipFile``` can then link to the Python documentation
+This extension can generate links to the documentation of objects in external
+projects, either explicitly through the :rst:role:`external` role, or as a
+fallback resolution for any other cross-reference.
+
+Usage for fallback resolution is simple: whenever Sphinx encounters a
+cross-reference that has no matching target in the current documentation set,
+it looks for targets in the external documentation sets configured in
+:confval:`intersphinx_mapping`.  A reference like
+``:py:class:`zipfile.ZipFile``` can then link to the Python documentation
 for the ZipFile class, without you having to specify where it is located
 exactly.
 
-When using the "new" format (see below), you can even force lookup in a foreign
-set by prefixing the link target appropriately.  A link like ``:ref:`comparison
-manual <python:comparisons>``` will then link to the label "comparisons" in the
-doc set "python", if it exists.
+When using the :rst:role:`external` role, you can force lookup to any external
+projects, and optionally to a specific external project.
+A link like ``:external:ref:`comparison manual <comparisons>``` will then link
+to the label "comparisons" in whichever configured external project, if it
+exists,
+and a link like ``:external+python:ref:`comparison manual <comparisons>``` will
+link to the label "comparisons" only in the doc set "python", if it exists.
 
 Behind the scenes, this works as follows:
 
@@ -30,8 +35,8 @@ Behind the scenes, this works as follows:
 
 * Projects using the Intersphinx extension can specify the location of such
   mapping files in the :confval:`intersphinx_mapping` config value.  The mapping
-  will then be used to resolve otherwise missing references to objects into
-  links to the other documentation.
+  will then be used to resolve both :rst:role:`external` references, and also
+  otherwise missing references to objects into links to the other documentation.
 
 * By default, the mapping file is assumed to be at the same location as the rest
   of the documentation; however, the location of the mapping file can also be
@@ -79,10 +84,10 @@ linking:
    at the same location as the base URI) or another local file path or a full
    HTTP URI to an inventory file.
 
-   The unique identifier can be used to prefix cross-reference targets, so that
+   The unique identifier can be used in the :rst:role:`external` role, so that
    it is clear which intersphinx set the target belongs to.  A link like
-   ``:ref:`comparison manual <python:comparisons>``` will link to the label
-   "comparisons" in the doc set "python", if it exists.
+   ``external:python+ref:`comparison manual <comparisons>``` will link to the
+   label "comparisons" in the doc set "python", if it exists.
 
    **Example**
 
@@ -162,21 +167,50 @@ linking:
 
    The default value is an empty list.
 
-   When a cross-reference without an explicit inventory specification is being
-   resolved by intersphinx, skip resolution if it matches one of the
-   specifications in this list.
+   When a non-:rst:role:`external` cross-reference is being resolved by
+   intersphinx, skip resolution if it matches one of the specifications in this
+   list.
 
    For example, with ``intersphinx_disabled_reftypes = ['std:doc']``
    a cross-reference ``:doc:`installation``` will not be attempted to be
-   resolved by intersphinx, but ``:doc:`otherbook:installation``` will be
-   attempted to be resolved in the inventory named ``otherbook`` in
+   resolved by intersphinx, but ``:external+otherbook:doc:`installation``` will
+   be attempted to be resolved in the inventory named ``otherbook`` in
    :confval:`intersphinx_mapping`.
    At the same time, all cross-references generated in, e.g., Python,
    declarations will still be attempted to be resolved by intersphinx.
 
-   If ``*`` is in the list of domains, then no references without an explicit
-   inventory will be resolved by intersphinx.
+   If ``*`` is in the list of domains, then no non-:rst:role:`external`
+   references will be resolved by intersphinx.
+
+Explicitly Reference External Objects
+-------------------------------------
+
+The Intersphinx extension provides the following role.
+
+.. rst:role:: external
+
+   .. versionadded:: 4.4
+
+   Use Intersphinx to perform lookup only in external projects, and not the
+   current project. Intersphinx still needs to know the type of object you
+   would like to find, so the general form of this role is to write the
+   cross-refererence as if the object is in the current project, but then prefix
+   it with ``:external``.
+   The two forms are then
+
+   - ``:external:domain:reftype:`target```,
+     e.g., ``:external:py:class:`zipfile.ZipFile```, or
+   - ``:external:reftype:`target```,
+     e.g., ``:external:doc:`installation```.
+
+   If you would like to constrain the lookup to a specific external project,
+   then the key of the project, as specified in :confval:`intersphinx_mapping`,
+   is added as well to get the two forms 
 
+   - ``:external+invname:domain:reftype:`target```,
+     e.g., ``:external+python:py:class:`zipfile.ZipFile```, or
+   - ``:external+invname:reftype:`target```,
+     e.g., ``:external+python:doc:`installation```.
 
 Showing all links of an Intersphinx mapping file
 ------------------------------------------------
diff --git a/sphinx/ext/intersphinx.py b/sphinx/ext/intersphinx.py
index 7f3588ade..2f8ab2588 100644
--- a/sphinx/ext/intersphinx.py
+++ b/sphinx/ext/intersphinx.py
@@ -26,15 +26,17 @@
 import concurrent.futures
 import functools
 import posixpath
+import re
 import sys
 import time
 from os import path
-from typing import IO, Any, Dict, List, Optional, Tuple
+from types import ModuleType
+from typing import IO, Any, Dict, List, Optional, Tuple, cast
 from urllib.parse import urlsplit, urlunsplit
 
 from docutils import nodes
-from docutils.nodes import Element, TextElement
-from docutils.utils import relative_path
+from docutils.nodes import Element, Node, TextElement, system_message
+from docutils.utils import Reporter, relative_path
 
 import sphinx
 from sphinx.addnodes import pending_xref
@@ -43,10 +45,13 @@ from sphinx.builders.html import INVENTORY_FILENAME
 from sphinx.config import Config
 from sphinx.domains import Domain
 from sphinx.environment import BuildEnvironment
+from sphinx.errors import ExtensionError
 from sphinx.locale import _, __
+from sphinx.transforms.post_transforms import ReferencesResolver
 from sphinx.util import logging, requests
+from sphinx.util.docutils import CustomReSTDispatcher, SphinxRole
 from sphinx.util.inventory import InventoryFile
-from sphinx.util.typing import Inventory, InventoryItem
+from sphinx.util.typing import Inventory, InventoryItem, RoleFunction
 
 logger = logging.getLogger(__name__)
 
@@ -466,6 +471,144 @@ def missing_reference(app: Sphinx, env: BuildEnvironment, node: pending_xref,
     return resolve_reference_detect_inventory(env, node, contnode)
 
 
+class IntersphinxDispatcher(CustomReSTDispatcher):
+    """Custom dispatcher for external role.
+
+    This enables :external:***:/:external+***: roles on parsing reST document.
+    """
+
+    def role(self, role_name: str, language_module: ModuleType, lineno: int, reporter: Reporter
+             ) -> Tuple[RoleFunction, List[system_message]]:
+        if len(role_name) > 9 and role_name.startswith(('external:', 'external+')):
+            return IntersphinxRole(role_name), []
+        else:
+            return super().role(role_name, language_module, lineno, reporter)
+
+
+class IntersphinxRole(SphinxRole):
+    # group 1: just for the optionality of the inventory name
+    # group 2: the inventory name (optional)
+    # group 3: the domain:role or role part
+    _re_inv_ref = re.compile(r"(\+([^:]+))?:(.*)")
+
+    def __init__(self, orig_name: str) -> None:
+        self.orig_name = orig_name
+
+    def run(self) -> Tuple[List[Node], List[system_message]]:
+        assert self.name == self.orig_name.lower()
+        inventory, name_suffix = self.get_inventory_and_name_suffix(self.orig_name)
+        if inventory and not inventory_exists(self.env, inventory):
+            logger.warning(__('inventory for external cross-reference not found: %s'),
+                           inventory, location=(self.env.docname, self.lineno))
+            return [], []
+
+        role_name = self.get_role_name(name_suffix)
+        if role_name is None:
+            logger.warning(__('role for external cross-reference not found: %s'), name_suffix,
+                           location=(self.env.docname, self.lineno))
+            return [], []
+
+        result, messages = self.invoke_role(role_name)
+        for node in result:
+            if isinstance(node, pending_xref):
+                node['intersphinx'] = True
+                node['inventory'] = inventory
+
+        return result, messages
+
+    def get_inventory_and_name_suffix(self, name: str) -> Tuple[Optional[str], str]:
+        assert name.startswith('external'), name
+        assert name[8] in ':+', name
+        # either we have an explicit inventory name, i.e,
+        # :external+inv:role:        or
+        # :external+inv:domain:role:
+        # or we look in all inventories, i.e.,
+        # :external:role:            or
+        # :external:domain:role:
+        inv, suffix = IntersphinxRole._re_inv_ref.fullmatch(name, 8).group(2, 3)
+        return inv, suffix
+
+    def get_role_name(self, name: str) -> Optional[Tuple[str, str]]:
+        names = name.split(':')
+        if len(names) == 1:
+            # role
+            default_domain = self.env.temp_data.get('default_domain')
+            domain = default_domain.name if default_domain else None
+            role = names[0]
+        elif len(names) == 2:
+            # domain:role:
+            domain = names[0]
+            role = names[1]
+        else:
+            return None
+
+        if domain and self.is_existent_role(domain, role):
+            return (domain, role)
+        elif self.is_existent_role('std', role):
+            return ('std', role)
+        else:
+            return None
+
+    def is_existent_role(self, domain_name: str, role_name: str) -> bool:
+        try:
+            domain = self.env.get_domain(domain_name)
+            if role_name in domain.roles:
+                return True
+            else:
+                return False
+        except ExtensionError:
+            return False
+
+    def invoke_role(self, role: Tuple[str, str]) -> Tuple[List[Node], List[system_message]]:
+        domain = self.env.get_domain(role[0])
+        if domain:
+            role_func = domain.role(role[1])
+
+            return role_func(':'.join(role), self.rawtext, self.text, self.lineno,
+                             self.inliner, self.options, self.content)
+        else:
+            return [], []
+
+
+class IntersphinxRoleResolver(ReferencesResolver):
+    """pending_xref node resolver for intersphinx role.
+
+    This resolves pending_xref nodes generated by :intersphinx:***: role.
+    """
+
+    default_priority = ReferencesResolver.default_priority - 1
+
+    def run(self, **kwargs: Any) -> None:
+        for node in self.document.traverse(pending_xref):
+            if 'intersphinx' not in node:
+                continue
+            contnode = cast(nodes.TextElement, node[0].deepcopy())
+            inv_name = node['inventory']
+            if inv_name is not None:
+                assert inventory_exists(self.env, inv_name)
+                newnode = resolve_reference_in_inventory(self.env, inv_name, node, contnode)
+            else:
+                newnode = resolve_reference_any_inventory(self.env, False, node, contnode)
+            if newnode is None:
+                typ = node['reftype']
+                msg = (__('external %s:%s reference target not found: %s') %
+                       (node['refdomain'], typ, node['reftarget']))
+                logger.warning(msg, location=node, type='ref', subtype=typ)
+                node.replace_self(contnode)
+            else:
+                node.replace_self(newnode)
+
+
+def install_dispatcher(app: Sphinx, docname: str, source: List[str]) -> None:
+    """Enable IntersphinxDispatcher.
+
+    .. note:: The installed dispatcher will uninstalled on disabling sphinx_domain
+              automatically.
+    """
+    dispatcher = IntersphinxDispatcher()
+    dispatcher.enable()
+
+
 def normalize_intersphinx_mapping(app: Sphinx, config: Config) -> None:
     for key, value in config.intersphinx_mapping.copy().items():
         try:
@@ -497,7 +640,9 @@ def setup(app: Sphinx) -> Dict[str, Any]:
     app.add_config_value('intersphinx_disabled_reftypes', [], True)
     app.connect('config-inited', normalize_intersphinx_mapping, priority=800)
     app.connect('builder-inited', load_mappings)
+    app.connect('source-read', install_dispatcher)
     app.connect('missing-reference', missing_reference)
+    app.add_post_transform(IntersphinxRoleResolver)
     return {
         'version': sphinx.__display_version__,
         'env_version': 1,
diff --git a/sphinx/util/docutils.py b/sphinx/util/docutils.py
index c3a6ff9e2..5ab766649 100644
--- a/sphinx/util/docutils.py
+++ b/sphinx/util/docutils.py
@@ -166,16 +166,14 @@ def patch_docutils(confdir: Optional[str] = None) -> Generator[None, None, None]
         yield
 
 
-class ElementLookupError(Exception):
-    pass
-
+class CustomReSTDispatcher:
+    """Custom reST's mark-up dispatcher.
 
-class sphinx_domains:
-    """Monkey-patch directive and role dispatch, so that domain-specific
-    markup takes precedence.
+    This replaces docutils's directives and roles dispatch mechanism for reST parser
+    by original one temporarily.
     """
-    def __init__(self, env: "BuildEnvironment") -> None:
-        self.env = env
+
+    def __init__(self) -> None:
         self.directive_func: Callable = lambda *args: (None, [])
         self.roles_func: Callable = lambda *args: (None, [])
 
@@ -189,13 +187,35 @@ class sphinx_domains:
         self.directive_func = directives.directive
         self.role_func = roles.role
 
-        directives.directive = self.lookup_directive
-        roles.role = self.lookup_role
+        directives.directive = self.directive
+        roles.role = self.role
 
     def disable(self) -> None:
         directives.directive = self.directive_func
         roles.role = self.role_func
 
+    def directive(self,
+                  directive_name: str, language_module: ModuleType, document: nodes.document
+                  ) -> Tuple[Optional[Type[Directive]], List[system_message]]:
+        return self.directive_func(directive_name, language_module, document)
+
+    def role(self, role_name: str, language_module: ModuleType, lineno: int, reporter: Reporter
+             ) -> Tuple[RoleFunction, List[system_message]]:
+        return self.role_func(role_name, language_module, lineno, reporter)
+
+
+class ElementLookupError(Exception):
+    pass
+
+
+class sphinx_domains(CustomReSTDispatcher):
+    """Monkey-patch directive and role dispatch, so that domain-specific
+    markup takes precedence.
+    """
+    def __init__(self, env: "BuildEnvironment") -> None:
+        self.env = env
+        super().__init__()
+
     def lookup_domain_element(self, type: str, name: str) -> Any:
         """Lookup a markup element (directive or role), given its name which can
         be a full name (with domain).
@@ -226,17 +246,20 @@ class sphinx_domains:
 
         raise ElementLookupError
 
-    def lookup_directive(self, directive_name: str, language_module: ModuleType, document: nodes.document) -> Tuple[Optional[Type[Directive]], List[system_message]]:  # NOQA
+    def directive(self,
+                  directive_name: str, language_module: ModuleType, document: nodes.document
+                  ) -> Tuple[Optional[Type[Directive]], List[system_message]]:
         try:
             return self.lookup_domain_element('directive', directive_name)
         except ElementLookupError:
-            return self.directive_func(directive_name, language_module, document)
+            return super().directive(directive_name, language_module, document)
 
-    def lookup_role(self, role_name: str, language_module: ModuleType, lineno: int, reporter: Reporter) -> Tuple[RoleFunction, List[system_message]]:  # NOQA
+    def role(self, role_name: str, language_module: ModuleType, lineno: int, reporter: Reporter
+             ) -> Tuple[RoleFunction, List[system_message]]:
         try:
             return self.lookup_domain_element('role', role_name)
         except ElementLookupError:
-            return self.role_func(role_name, language_module, lineno, reporter)
+            return super().role(role_name, language_module, lineno, reporter)
 
 
 class WarningStream:
diff --git a/tests/roots/test-ext-intersphinx-role/conf.py b/tests/roots/test-ext-intersphinx-role/conf.py
new file mode 100644
index 000000000..a54f5c2ad
--- /dev/null
+++ b/tests/roots/test-ext-intersphinx-role/conf.py
@@ -0,0 +1,3 @@
+extensions = ['sphinx.ext.intersphinx']
+# the role should not honor this conf var
+intersphinx_disabled_reftypes = ['*']
diff --git a/tests/roots/test-ext-intersphinx-role/index.rst b/tests/roots/test-ext-intersphinx-role/index.rst
new file mode 100644
index 000000000..58edb7a1a
--- /dev/null
+++ b/tests/roots/test-ext-intersphinx-role/index.rst
@@ -0,0 +1,44 @@
+- ``module1`` is only defined in ``inv``:
+  :external:py:mod:`module1`
+
+.. py:module:: module2
+
+- ``module2`` is defined here and also in ``inv``, but should resolve to inv:
+  :external:py:mod:`module2`
+
+- ``module3`` is not defined anywhere, so should warn:
+  :external:py:mod:`module3`
+
+.. py:module:: module10
+
+- ``module10`` is only defined here, but should still not be resolved to:
+  :external:py:mod:`module10`
+
+- a function in inv:
+  :external:py:func:`module1.func`
+- a method, but with old style inventory prefix, which shouldn't work:
+  :external:py:meth:`inv:Foo.bar`
+- a non-existing role:
+  :external:py:nope:`something`
+
+.. default-domain:: cpp
+
+- a type where the default domain is used to find the role:
+  :external:type:`std::uint8_t`
+- a non-existing role in default domain:
+  :external:nope:`somethingElse`
+
+- two roles in ``std`` which can be found without a default domain:
+
+  - :external:doc:`docname`
+  - :external:option:`ls -l`
+
+
+- a function with explicit inventory:
+  :external+inv:c:func:`CFunc`
+- a class with explicit non-existing inventory, which also has upper-case in name:
+  :external+invNope:cpp:class:`foo::Bar`
+
+
+- explicit title:
+  :external:cpp:type:`FoonsTitle <foons>`
diff --git a/tests/test_ext_intersphinx.py b/tests/test_ext_intersphinx.py
index 7f369e9a3..b2ad8afe5 100644
--- a/tests/test_ext_intersphinx.py
+++ b/tests/test_ext_intersphinx.py
@@ -524,3 +524,48 @@ def test_inspect_main_url(capsys):
     stdout, stderr = capsys.readouterr()
     assert stdout.startswith("c:function\n")
     assert stderr == ""
+
+
+@pytest.mark.sphinx('html', testroot='ext-intersphinx-role')
+def test_intersphinx_role(app, warning):
+    inv_file = app.srcdir / 'inventory'
+    inv_file.write_bytes(inventory_v2)
+    app.config.intersphinx_mapping = {
+        'inv': ('http://example.org/', inv_file),
+    }
+    app.config.intersphinx_cache_limit = 0
+    app.config.nitpicky = True
+
+    # load the inventory and check if it's done correctly
+    normalize_intersphinx_mapping(app, app.config)
+    load_mappings(app)
+
+    app.build()
+    content = (app.outdir / 'index.html').read_text()
+    wStr = warning.getvalue()
+
+    html = '<a class="reference external" href="http://example.org/{}" title="(in foo v2.0)">'
+    assert html.format('foo.html#module-module1') in content
+    assert html.format('foo.html#module-module2') in content
+    assert "WARNING: external py:mod reference target not found: module3" in wStr
+    assert "WARNING: external py:mod reference target not found: module10" in wStr
+
+    assert html.format('sub/foo.html#module1.func') in content
+    assert "WARNING: external py:meth reference target not found: inv:Foo.bar" in wStr
+
+    assert "WARNING: role for external cross-reference not found: py:nope" in wStr
+
+    # default domain
+    assert html.format('index.html#std_uint8_t') in content
+    assert "WARNING: role for external cross-reference not found: nope" in wStr
+
+    # std roles without domain prefix
+    assert html.format('docname.html') in content
+    assert html.format('index.html#cmdoption-ls-l') in content
+
+    # explicit inventory
+    assert html.format('cfunc.html#CFunc') in content
+    assert "WARNING: inventory for external cross-reference not found: invNope" in wStr
+
+    # explicit title
+    assert html.format('index.html#foons') in content