Add the "See Also" section to the TryExamples directive's trailing section ignore patterns

[agriyakhetarpal]

Description

This PR is a bug fix for a problem I noticed in sympy/sympy#27419. SymPy uses a vendored version of numpydoc that was added twelve years back based on https://github.com/numpy/numpydoc; both have diverged in implementation and complexity since then, and SymPy's numpydoc doesn't enforce a particular order for each section, unlike the current numpydoc, which can automatically order/reorder sections. This causes the following behaviour:

Tap to show "Before"

where the See Also section percolates into the notebook. With this fix in place applied, we have

Tap to show "After"

where the section no longer appears in the notebook, and we stop at the end of the Examples section, as intended.

Additional context

It would be nice to update the SymPy documentation infrastructure to use the latest numpydoc so that the documentation is consistent with that for other packages. However, considering that its custom numpydoc has had multiple stylistic deviations by now, I've left it to be a follow-up task for me after the PR linked above gets merged. So, supporting SymPy in a special case in this manner doesn't sound too bad to me, considering that this change is trivial.

[steppi]

"See Also" is included in _non_example_docstring_section_headers, so I don't think the issue can hinge just on the ability of "See Also" sections to appear after an "Examples" sections in older numpydocs versions. It looks like in SymPy, after numpydoc preprocessing, the line for the See Also section is ".. seealso::", but the existing logic expects it to look like ".. rubric:: See Also". I think the real issue is that the older numpydoc version uses a different format for the directives that get inserted during processing. It just so happens that SymPy has a "See Also" section after each "Examples" section, but I think any section appearing afterwards would fail to be detected.

If we want to support older numpydoc versions, then I think the principled solution would then be to either:

1. Add the format used by older numpydoc versions to _next_section_pattern
   or
2. Detect the numpydoc version at runtime, and have _next_section_start_pattern depend on that version

update: or it could even be that the directives even for newer numpydoc versions are different from what I expected, but I had it right for every section that can appear after "Examples" with newer numpydoc versions.

[agriyakhetarpal]

It just so happens that SymPy has a "See Also" section after each "Examples" section, but I think any section appearing afterwards would fail to be detected.

Ah, you're right – thanks for pointing that out! I think it would be nice to support older numpydoc versions in whatever manner we can, but I think option 2) is not possible with SymPy's numpydoc at the moment, as it is present as a custom Sphinx extension in docs/sphinxext/ across two files, and not as an installable package fetched from PyPI or a separate location that we can compare or detect versions for. So we are left with 1), which I've done here.

[agriyakhetarpal]

Actually, the See Also section seems to be a case that has been handled specially: https://github.com/sympy/sympy/blob/3c33a8283cd02e2e012b57435bd6212ed915a0bc/doc/ext/docscrape_sphinx.py#L155-L161

And it doesn't seem to be the case for the "Notes", "Examples", or "References" sections: https://github.com/sympy/sympy/blob/3c33a8283cd02e2e012b57435bd6212ed915a0bc/doc/ext/docscrape.py#L424, which are not included in Try Ex

I haven't dived into the code in detail, but I infer that the difference is because the "See Also" section is an admonition in SymPy's numpydoc, and not a rubric. I'd suggest that we stick with the current approach for now, and add any other cases here if someone notices them, until I can migrate SymPy to the upstream numpydoc and remove their implementation (that is, I'm short of ideas, unless you have some).

[steppi]

If this is working, it's fine to do this hack to get SymPy working specifically, but the comment is incorrect and should be updated. It's not about the ordering but because of the form the directive takes in sympy's version of numpydocs.

[agriyakhetarpal]

Yes, makes sense to me. I hope 5f187f3 addresses it!

[steppi]

This is again not quite right. If you check line 311, it also checks for admonitions. The regex rf"\.\. (rubric|admonition)::\s*{header} will match .. followed by one of rubric or admonition, followed by any number of whitespace, followed by one of the headers in _non_example_docstring_headers.

But in at least SymPy, the See Also section doesn't get a directive like .. admonition:: See Also but just .. seealso:. What I'm curious about is, for current versions of numpydocs, does a See Also section get the directive .. admonition:: See Also like I assumed, or was I incorrect about this?

Since the above logic does check for See Also sections, it can't be relevant that the See Also section shows up for old numpydoc versions.

Suggested change

	# The See Also section shows up for old numpydoc versions such as that
	# of SymPy where it is accounted for as an admonition and not a rubric,
	# therefore we check for it as a special case for now.
	# SymPy uses a custom numpydoc version which has a different pattern for
	# the generated directives. Rather than investigating further, take advantage
	# of See Also being the only section appearing after Examples in SymPy and
	# hardcode the directive that appears in SymPy. This can be removed after
	# SymPy is updated to use regular numpydoc.

[steppi]

At some point someone should sit down and figure out what the generated directives can actually look like for each section header in mainline numpydoc.

[agriyakhetarpal]

I spent some time debugging this and while I couldn't get a definite answer about what happens where in the code, here's what I have realised –

There are two things at play here:

1. There is a a .. seealso:: directive that's native to Sphinx, and a "See Also" section that upstream numpydoc can interpret and convert to a rubric/admonition, which we correctly parse and escape with TryExamples' _next_section_pattern if we retain compatibility with upstream numpydoc.
2. However, SymPy's numpydoc is dated and can't do this for SymPy's documentation, and thus we don't get to catch it.

Therefore, a "See Also" section in the documentation for SciPy for an example, will caught in numpydoc, and this problem isn't specific to just .. seealso::, all native Sphinx-based directives won't get caught. It's just that this hasn't been reported as a bug before because of the following reasons:

A. most projects don't put anything after "Examples", and only SymPy that we know of (so far) has done so
B. Upstream numpydoc does indeed reorder the sections to some extent (which was my original comment a few commits ago, so I was right). Even if a project does add a section after "Examples", it gets caught by the pattern or re-ordered before it's rendered in the HTML.

As a confirmation for point A, I tried another Sphinx directive from the page I linked above, such as .. versionadded::, and here we go:

SymPy documentation screenshot with erroneous versionadded directive

The same experiment with NumPy confirms point B. For example, the docstring below:

NumPy's exceptions.AxisError docstring (modified)

"""
    Examples
    --------
    >>> import numpy as np
    >>> array_1d = np.arange(10)
    >>> np.cumsum(array_1d, axis=1)
    Traceback (most recent call last):
      ...
    numpy.exceptions.AxisError: axis 1 is out of bounds for array of dimension 1

    Negative axes are preserved:

    >>> np.cumsum(array_1d, axis=-2)
    Traceback (most recent call last):
      ...
    numpy.exceptions.AxisError: axis -2 is out of bounds for array of dimension 1

    The class constructor generally takes the axis and arrays'
    dimensionality as arguments:

    >>> print(np.exceptions.AxisError(2, 1, msg_prefix='error'))
    error: axis 2 is out of bounds for array of dimension 1

    Alternatively, a custom exception message can be passed:

    >>> print(np.exceptions.AxisError('Custom error message'))
    Custom error message

    See Also
    --------

    numpy.core._exceptions._AxisError : The internal version of this exception.

    Attributes
    ----------
    axis : int, optional
        The out of bounds axis or ``None`` if a custom exception
        message was provided. This should be the axis as passed by
        the user, before any normalization to resolve negative indices.

        .. versionadded:: 1.22
    ndim : int, optional
        The number of array dimensions or ``None`` if a custom exception
        message was provided.

        .. versionadded:: 1.22
"""

where the "See Also" section is after the "Examples" section – where we could assume that it would be ordered in the same manner. However, generating the NumPy docs, it shows up before the examples:

Screenshot 1 from NumPy documentation

But, if you were to add the .. versionadded:: directive or similar ones for NumPy, they are not reordered. I added a .. versionchanged:: Agriya debugging directive after the Examples section, and this is what we get:

NumPy documentation screenshots 2 and 3 with a native Sphinx-based directive after the Examples section

The HTML

and the interactive notebook

[agriyakhetarpal]

We'll need to add almost all of the directives listed under https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#paragraph-level-markup to be able to restrict them from percolating into the notebooks; I don't think it makes sense to leave it with just .. seealso especially if others like ..versionadded are also included with both SymPy's numpydoc and the upstream numpydoc.

I used this command

from docutils.parsers.rst import directives
all_directives = list(directives._directive_registry.keys())

to get a list:

['attention',
 'caution',
 'code',
 'danger',
 'error',
 'important',
 'note',
 'tip',
 'hint',
 'warning',
 'admonition',
 'sidebar',
 'topic',
 'line-block',
 'parsed-literal',
 'math',
 'rubric',
 'epigraph',
 'highlights',
 'pull-quote',
 'compound',
 'container',
 'table',
 'csv-table',
 'list-table',
 'image',
 'figure',
 'contents',
 'sectnum',
 'header',
 'footer',
 'target-notes',
 'meta',
 'raw',
 'include',
 'replace',
 'unicode',
 'class',
 'role',
 'default-role',
 'title',
 'date',
 'restructuredtext-test-directive']

and we know some Sphinx-specific directives (outside docutils) manually that aren't present in this list:

['seealso',
 'versionadded',
 'versionchanged',
 'deprecated',
 'versionremoved',
 'index'
]

Only a subset of them will need to be added to the regex pattern, I assume?

(Edit: added a subset, please keep reading below)

[agriyakhetarpal]

I remembered this numpydoc PR from last year that was xref-ed from the NumPy PR where I had reported some incorrect ordering of the Attributes section, as it was showing up after the Examples section. It sheds some light on how the sections are ordered/reordered: numpy/numpydoc#571.

The order preservation is mandated here: https://github.com/numpy/numpydoc/blob/6d5eb428b8acea6ced4d86d086e36daf3f03ffbf/numpydoc/validate.py#L28-L41, and via an error message in https://github.com/numpy/numpydoc/blob/6d5eb428b8acea6ced4d86d086e36daf3f03ffbf/numpydoc/validate.py#L61. We know that dictionaries are ordered in Python post 3.7.

Comparing the relevant code with SymPy's numpydoc: https://github.com/sympy/sympy/blob/3c33a8283cd02e2e012b57435bd6212ed915a0bc/doc/ext/docscrape_sphinx.py#L208-L240, not only are the headings and sections different, but also that upstream numpydoc has a structural templating mechanism thingy where an order is enforced, and SymPy's version explicitly calls _str_see_also at the point where it wants the section to appear.

So, to answer your question (quoting it below) with all I've mentioned:

But in at least SymPy, the See Also section doesn't get a directive like .. admonition:: See Also but just .. seealso:. What I'm curious about is, for current versions of numpydocs, does a See Also section get the directive .. admonition:: See Also like I assumed, or was I incorrect about this?

Since the above logic does check for See Also sections, it can't be relevant that the See Also section shows up for old numpydoc versions.

If upstream numpydoc had not enforced the order, we would have also faced this SymPy-specific problem in NumPy. It is about the ordering of the sections rather than what the sections are themselves (whether Sphinx directives or named sections in numpydoc).

Upstream numpydoc also has https://github.com/numpy/numpydoc/blob/6d5eb428b8acea6ced4d86d086e36daf3f03ffbf/numpydoc/validate.py#L24-L27 to run checks version-changed directives like the above, but it does not enforce an order for them as they are not a part of standard sections but are directives.

So, I think we should indeed add at least a few of these directives to the patterns we need to ignore – maybe just the most commonly used ones from the above list will work. This would help in both getting SymPy to play nicely, and in the scenario that some docstrings are incorrectly added to any other projects that use the modern numpydoc, we don't let that mistake in with the interactive examples.

[agriyakhetarpal]

At the same time, if someone were to put a directive in the middle of a doctest, that could cause a bug report for us because the doctest would end midway... but I feel we're technically correct and we could say "works as intended" and close it.

[agriyakhetarpal]

I've added a minimal set of directives to halt at in commit 79e864e and this shouldn't cause any problematic behaviour with most websites, while also getting us working with SymPy.