Error when returning a nodes.section in a custom directive

[loichuder]

I have a very simple Sphinx extension that implement a custom directive that returns a new section:

from docutils.parsers.rst import Directive
from docutils import nodes


class TestDirective(Directive):
    def run(self):
        new_section = nodes.section()
        new_section += nodes.title(text="Title")
        return [new_section]

def setup(app):
    app.add_directive("test", TestDirective)

But whenever I try to include the directive in a Sphinx project:

test
====

.. test:: 

I get the following error:

Extension error (sphinx.environment.collectors.toctree):
Handler <bound method TocTreeCollector.process_doc of <sphinx.environment.collectors.toctree.TocTreeCollector object at 0x7f8e3ebed390>> for event 'doctree-read' threw an exception (exception: list index out of range)

I am still very new to docutils and custom Sphinx extensions so I guess I am doing something wrong but I don't have the skills to really dig deeper.

Ps: I am using Sphinx 8.1.3 but I get the same error with Sphinx 7.x

[AA-Turner]

I'd first suggest using SphinxDirective, from sphinx.util.docutils. Can you paste a full reproducer?

A

[loichuder]

Thanks for your answer @AA-Turner

I tried with SphinxDirective but to no avail, I get the same error. Here is the full code I used:

# conf.py

from sphinx.util.docutils import SphinxDirective
from docutils import nodes

class TestDirective(SphinxDirective):
    def run(self):
        new_section = nodes.section()
        new_section += nodes.title(
            text="Title",
        )
        return [new_section]


def setup(app):
    app.add_directive("test", TestDirective)

.. index.rst

Hello
=====

.. test::

I had these two files in the same folder and ran the following command inside the folder:

sphinx-build . html

[loichuder]

If that could help, I found a workaround. If I specify ids as a arg of my new section in TestDirective :

new_section = nodes.section(ids=[''])

The error does not show up and the directive produces the expected result.

But I have no clue why 😅

[AA-Turner]

I would suggest the following:

from docutils import nodes
from sphinx.util.docutils import SphinxDirective


class TestDirective(SphinxDirective):
    def run(self):
        new_section = nodes.section()
        new_section += nodes.title('Title', 'Title')
        self.state.document.note_implicit_target(new_section, new_section)
        return [new_section]


def setup(app):
    app.add_directive('test', TestDirective)

Note note_implicit_target, which both creates an internal reference target to your section, and sets the node ID.

You may find the relevant Docutils code useful here, as this is what happens for the Docutils parser to create a section:

https://github.com/docutils/docutils/blob/f704fb58904d62bf5c6d7db82a9628f29e4ed246/docutils/docutils/parsers/rst/states.py#L383-L392

However, I would counsel caution -- creating section nodes in a directive can easily lead to an incoherent document tree. See the admonition in parse_content_to_nodes():

Note that this option bypasses Docutils’ usual checks on doctree structure, and misuse of this option can lead to an incoherent doctree. In Docutils, section nodes should only be children of Structural nodes, which includes document, section, and sidebar nodes.

See @chrisjsewell's article on nested headings, and comment here for further context.

A

[chrisjsewell]

Yeh, this is definitely not advised 😅

Docutils nest things under the sections as it progresses the parse (as opposed to doing the nesting in a post-process), and e.g. handles how to exit out of a section into a parent, by tracking current nesting levels.

Unless you are handling all of this, then you are not going to get the correct outcome
(not to speak of how inoperable this is with MyST 🫠)

[loichuder]

Thank you both. The article on nested headings was indeed very enlightening! I have one last question if you don't mind, to be sure that I understood correctly.

So not only it is not recommended to create sections in a directive, but it is not recommended to create titles in directives as well ?

Since the directive may be placed in a admonition, leading to the flawed document structure described in this part of the article, right ?