IndexError: pop from empty list when making mesa docs witn Sphinx 8.2 #13407

chennin · 2025-03-03T14:52:54Z

Describe the bug

Hello! I'm just an Arch Linux user trying to compile mesa. Generating its HTML docs fails with Sphinx 8.2.1 and 8.2.3; and succeeds with 8.1.3. I'm not sure where exactly the issue lies - I have opened a mesa bug too. I've replicated on mesa 24.3.4, 25.0.0, and main 264d8a676645da829495bafbc477dea5baa52fa8.

Attached log:
sphinx-err-g1gudqzy.log

Impact: Again I am not an Arch maintainer - but I suspect this will become quite relevant to Arch when Mesa 25.0.1 is released and Arch proceeds to try to build it.

How to Reproduce

cd /tmp
git clone https://gitlab.freedesktop.org/mesa/mesa.git
docker run --rm -it -v $PWD/mesa:/data archlinux:latest

# In docker:
pacman -Syu --noconfirm base-devel clang   expat   gcc-libs   glibc   libdrm   libelf   libglvnd   libpng   libva   libvdpau   libx11   libxcb   libxext   libxml2   libxrandr   libxshmfence   libxxf86vm   llvm   llvm-libs   lm_sensors   rust   spirv-llvm-translator   spirv-tools   systemd-libs   vulkan-icd-loader   wayland   xcb-util-keysyms   zlib   zstd   cbindgen   clang   cmake   elfutils   glslang   libclc   meson   python-mako   python-packaging   python-ply   python-yaml   rust-bindgen   wayland-protocols   xorgproto   valgrind   directx-headers   python-sphinx   python-sphinx-hawkmoth
cd /data
meson setup builddir/ -D html-docs=enabled
# IF YOU WANT TO WAIT FOR COMPILATION:
meson compile -C builddir/
# IF YOU JUST WANT TO RUN SPHINX:
/usr/bin/sphinx-build -vvT -b html -q -Ddepfile=docs/docs.d docs docs/html # what meson runs plus verbose flags

# The above fails. Downgrade Sphinx:
pacman -U --noconfirm https://archive.archlinux.org/packages/p/python-sphinx/python-sphinx-8.1.3-2-any.pkg.tar.zst
/usr/bin/sphinx-build -vvT -b html -q -Ddepfile=docs/docs.d docs docs/html # what meson runs plus verbose flags
# OR
meson compile -C builddir/
# This succeeds.

Environment Information

I've replicated both on base Arch (fully update) and for this report, in docker on an Ubuntu host (thus the kernel is lower than Arch's native).

Platform:              linux; (Linux-6.8.0-54-generic-x86_64-with-glibc2.41)
Python version:        3.13.2 (main, Feb  5 2025, 08:05:21) [GCC 14.2.1 20250128])
Python implementation: CPython
Sphinx version:        8.2.3
Docutils version:      0.21.2
Jinja2 version:        3.1.5
Pygments version:      2.19.1

Sphinx extensions

Additional context

Mesa bug report https://gitlab.freedesktop.org/mesa/mesa/-/issues/12725

The text was updated successfully, but these errors were encountered:

AA-Turner · 2025-03-03T16:14:58Z

@chennin can you paste the full failing log, when running with 8.2.3, please?

A

chennin · 2025-03-03T16:18:07Z

Sure, I assume you mean this more than the generated full traceback log file?

log

[root@3c0b14d25242 data]#       /usr/bin/sphinx-build -vvT -b html -q -Ddepfile=docs/docs.d docs docs/html # what meson runs plus verbose flags


Versions
========

* Platform:         linux; (Linux-6.8.0-54-generic-x86_64-with-glibc2.41)
* Python version:   3.13.2 (CPython)
* Sphinx version:   8.2.3
* Docutils version: 0.21.2
* Jinja2 version:   3.1.5
* Pygments version: 2.19.1

Last Messages
=============

    writing output... [  1%]
    ci/docker

    [app] emitting event: 'doctree-resolved'(<document: <substitution_definition "in"...><substitution_definitio ...>, 'ci/docker')
    [app] emitting event: 'html-page-context'('ci/docker', 'page.html', {'embedded': False, 'project': 'The Mesa 3D Graphics Library', 'release': 'latest', 'version': 'latest', 'last_updated': None, 'copyright': '1995-2018, Brian Paul', 'master_doc': 'index', 'root_doc': 'index', 'use_opensearch': '', 'docstitle': 'The Mesa 3D Graphics Library latest documentation', 'shorttitle': 'The Mesa 3D Graphics Library latest documentation', 'show_copyright': False, 'show_search_summary': True, 'show_sphinx': True, 'has_source': False, 'show_source': True, 'sourcelink_suffix': '.txt', 'file_suffix': '.html', 'link_suffix': '.html', 'script_files': [<sphinx.builders.html._assets._JavaScript object at 0x7266cff5ef90>, <sphinx.builders.html._assets._JavaScript object at 0x7266ceda6350>, <sphinx.builders.html._assets._JavaScript object at 0x7266ceda6490>], 'language': 'en', 'css_files': [<sphinx.builders.html._assets._CascadingStyleSheet object at 0x7266ceda60d0>, <sphinx.builders.html._assets._CascadingStyleSheet object at 0x7266ceda6210>, <sphinx.builders.html._assets._CascadingStyleSheet object at 0x7266cec30c30>, <sphinx.builders.html._assets._CascadingStyleSheet object at 0x7266cec30e90>], 'sphinx_version': '8.2.3', 'sphinx_version_tuple': (8, 2, 3, 'final', 0), 'docutils_version_info': (0, 21, 2, 'final', 0), 'styles': ['screen.css'], 'rellinks': [('genindex', 'General Index', 'I', 'index'), ('ci/local-traces', 'Running traces on a local machine', 'N', 'next'), ('ci/LAVA', 'LAVA CI', 'P', 'previous')], 'builder': 'html', 'parents': [{'link': 'index.html', 'title': 'Continuous Integration'}], 'logo_url': '', 'logo_alt': 'Logo of The Mesa 3D Graphics Library', 'favicon_url': 'favicon.ico', 'html5_doctype': True, 'theme_nosidebar': 'false', 'theme_sidebarwidth': '230', 'theme_body_min_width': '360', 'theme_body_max_width': '800', 'theme_navigation_with_keys': 'False', 'theme_enable_search_shortcuts': 'True', 'theme_globaltoc_collapse': 'true', 'theme_globaltoc_includehidden': 'false', 'theme_globaltoc_maxdepth': '', 'pagename': 'ci/docker', 'current_page_name': 'ci/docker', 'encoding': 'utf-8', 'pageurl': None, 'pathto': <function StandaloneHTMLBuilder.handle_page.<locals>.pathto at 0x7266a2359a80>, 'hasdoc': <function StandaloneHTMLBuilder.handle_page.<locals>.hasdoc at 0x7266a235ba60>, 'toctree': <function StandaloneHTMLBuilder.handle_page.<locals>.<lambda> at 0x7266a2358720>, 'sidebars': ['localtoc.html', 'relations.html', 'sourcelink.html', 'searchbox.html'], 'prev': {'link': 'LAVA.html', 'title': 'LAVA CI'}, 'next': {'link': 'local-traces.html', 'title': 'Running traces on a local machine'}, 'title': 'Docker CI', 'meta': {}, 'body': '<section id="docker-ci">\n<h1>Docker CI<a class="headerlink" href="#docker-ci" title="Link to this heading">¶</a></h1>\n<p>For LLVMpipe and Softpipe CI, we run tests in a container containing\nVK-GL-CTS, on the shared GitLab runners provided by <a class="reference external" href="https://www.freedesktop.org">freedesktop</a></p>\n<section id="software-architecture">\n<h2>Software architecture<a class="headerlink" href="#software-architecture" title="Link to this heading">¶</a></h2>\n<p>The Docker containers are rebuilt using the shell scripts under\n.gitlab-ci/container/ when the FDO_DISTRIBUTION_TAG changes in\n.gitlab-ci.yml. The resulting images are around 1 GB, and are\nexpected to change approximately weekly (though an individual\ndeveloper working on them may produce many more images while trying to\ncome up with a working MR!).</p>\n<p>gitlab-runner is a client that polls gitlab.freedesktop.org for\navailable jobs, with no inbound networking requirements.  Jobs can\nhave tags, so we can have DUT-specific jobs that only run on runners\nwith that tag marked in the GitLab UI.</p>\n<p>Since dEQP takes a long time to run, we mark the job as “parallel” at\nsome level, which spawns multiple jobs from one definition, and then\ndeqp-runner.sh takes the corresponding fraction of the test list for\nthat job.</p>\n<p>To reduce dEQP runtime (or avoid tests with unreliable results), a\ndeqp-runner.sh invocation can provide a list of tests to skip.  If\nyour driver is not yet conformant, you can pass a list of expected\nfailures, and the job will only fail on tests that aren’t listed (look\nat the job’s log for which specific tests failed).</p>\n</section>\n<section id="dut-requirements">\n<h2>DUT requirements<a class="headerlink" href="#dut-requirements" title="Link to this heading">¶</a></h2>\n<p>In addition to the general <a class="reference internal" href="index.html#ci-job-user-expectations"><span class="std std-ref">CI job user expectations</span></a>, using\nDocker requires:</p>\n<ul class="simple">\n<li><p>DUTs must have a stable kernel and GPU reset (if applicable).</p></li>\n</ul>\n<p>If the system goes down during a test run, that job will eventually\ntime out and fail (default 1 hour).  However, if the kernel can’t\nreliably reset the GPU on failure, bugs in one MR may leak into\nspurious failures in another MR.  This would be an unacceptable impact\non Mesa developers working on other drivers.</p>\n<ul class="simple">\n<li><p>DUTs must be able to run Docker</p></li>\n</ul>\n<p>The Mesa gitlab-runner based test architecture is built around Docker,\nso that we can cache the Debian package installation and CTS build\nstep across multiple test runs.  Since the images are large and change\napproximately weekly, the DUTs also need to be running some script to\nprune stale Docker images periodically in order to not run out of disk\nspace as we rev those containers (perhaps <a class="reference external" href="https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2980#note_169233611">this script</a>).</p>\n<p>Note that Docker doesn’t allow containers to be stored on NFS, and\ndoesn’t allow multiple Docker daemons to interact with the same\nnetwork block device, so you will probably need some sort of physical\nstorage on your DUTs.</p>\n<ul class="simple">\n<li><p>DUTs must be public</p></li>\n</ul>\n<p>By including your device in .gitlab-ci.yml, you’re effectively letting\nanyone on the internet run code on your device.  Docker containers may\nprovide some limited protection, but how much you trust that and what\nyou do to mitigate hostile access is up to you.</p>\n<ul class="simple">\n<li><p>DUTs must expose the DRI device nodes to the containers.</p></li>\n</ul>\n<p>Obviously, to get access to the HW, we need to pass the render node\nthrough.  This is done by adding <code class="docutils literal notranslate"><span class="pre">devices</span> <span class="pre">=</span> <span class="pre">[&quot;/dev/dri&quot;]</span></code> to the\n<code class="docutils literal notranslate"><span class="pre">runners.docker</span></code> section of /etc/gitlab-runner/config.toml.</p>\n</section>\n</section>\n', 'metatags': '<meta name="viewport" content="width=device-width, initial-scale=1" />\n', 'sourcename': '', 'toc': '<ul>\n<li><a class="reference internal" href="#">Docker CI</a><ul>\n<li><a class="reference internal" href="#software-architecture">Software architecture</a></li>\n<li><a class="reference internal" href="#dut-requirements">DUT requirements</a></li>\n</ul>\n</li>\n</ul>\n', 'display_toc': True, 'page_source_suffix': '.rst', 'has_maths_elements': False, 'content_root': '../', 'css_tag': <function StandaloneHTMLBuilder.handle_page.<locals>.css_tag at 0x7266a2358540>, 'js_tag': <function StandaloneHTMLBuilder.handle_page.<locals>.js_tag at 0x7266a23593a0>}, <document: <substitution_definition "in"...><substitution_definitio ...>)
    writing output... [  2%]
    ci/index

    [app] emitting event: 'doctree-resolved'(<document: <substitution_definition "in"...><substitution_definitio ...>, 'ci/index')
    [app] emitting event: 'build-finished'(IndexError('pop from empty list'),)

Loaded Extensions
=================

* sphinx.ext.mathjax (8.2.3)
* alabaster (1.0.0)
* sphinxcontrib.applehelp (2.0.0)
* sphinxcontrib.devhelp (2.0.0)
* sphinxcontrib.htmlhelp (2.1.0)
* sphinxcontrib.serializinghtml (2.0.0)
* sphinxcontrib.qthelp (2.0.0)
* bootstrap (unknown version)
* depfile (unknown version)
* formatting (unknown version)
* hawkmoth (0.19.0)
* nir (unknown version)
* redirects (unknown version)
* sphinx.ext.graphviz (8.2.3)

Traceback
=========

    Traceback (most recent call last):
      File "/usr/lib/python3.13/site-packages/sphinx/cmd/build.py", line 432, in build_main
        app.build(args.force_all, args.filenames)
        ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/application.py", line 426, in build
        self.builder.build_update()
        ~~~~~~~~~~~~~~~~~~~~~~~~~^^
      File "/usr/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 375, in build_update
        self.build(
        ~~~~~~~~~~^
            to_build,
            ^^^^^^^^^
        ...<2 lines>...
            method='update',
            ^^^^^^^^^^^^^^^^
        )
        ^
      File "/usr/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 454, in build
        self.write(docnames, updated_docnames, method)
        ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 735, in write
        self.write_documents(docnames)
        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 749, in write_documents
        self._write_serial(sorted_docnames)
        ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 768, in _write_serial
        self.write_doc(docname, doctree)
        ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/builders/html/__init__.py", line 670, in write_doc
        self.docwriter.write(doctree, destination)
        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/docutils/writers/__init__.py", line 80, in write
        self.translate()
        ~~~~~~~~~~~~~~^^
      File "/usr/lib/python3.13/site-packages/sphinx/writers/html.py", line 36, in translate
        self.document.walkabout(visitor)
        ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/docutils/nodes.py", line 186, in walkabout
        if child.walkabout(visitor):
           ~~~~~~~~~~~~~~~^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/docutils/nodes.py", line 186, in walkabout
        if child.walkabout(visitor):
           ~~~~~~~~~~~~~~~^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/docutils/nodes.py", line 186, in walkabout
        if child.walkabout(visitor):
           ~~~~~~~~~~~~~~~^^^^^^^^^
      File "/usr/lib/python3.13/site-packages/docutils/nodes.py", line 199, in walkabout
        visitor.dispatch_departure(self)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/util/docutils.py", line 783, in dispatch_departure
        method(node)
        ~~~~~~^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/writers/html5.py", line 845, in depart_note
        self.depart_admonition(node)
        ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
      File "/usr/lib/python3.13/site-packages/sphinx/writers/html5.py", line 383, in depart_admonition
        self.body.append(self.context.pop())
                         ~~~~~~~~~~~~~~~~^^
    IndexError: pop from empty list


The full traceback has been saved in:
/tmp/sphinx-err-l6onvpdy.log

To report this error to the developers, please open an issue at <https://github.com/sphinx-doc/sphinx/issues/>. Thanks!
Please also report this if it was a user error, so that a better error message can be provided next time.

AA-Turner · 2025-03-03T16:24:26Z

Thanks, that indicates that something has gone wrong with the abstract node tree whilst processing a .. note:: directive.

I know nearly nothing about mesa or Arch Linux, so I'd be grateful for your help in minimising to a reproducible example. What might be fastest is sucessively deleting half of the documentation until the error occurs/doesn't, in a sort of manual bisection process. The ideal minimum state we want is a conf.py with as few extensions as possible, and a single index.rst with the markup triggering the failure.

chennin · 2025-03-03T18:02:19Z

I'm also unfamiliar with mesa, and with sphinx, but! I found that if I only comment out bootstrap from the extension list, then the build succeeds.

That seems to be a file added by mesa based on pydata.

AA-Turner · 2025-03-03T19:49:33Z

Thank you! The error in this case is likely in the custom bootstrap extension. I don't have much time to investigate myself, though, sadly.

A

…issue - Added code to disable bootstrap extension in mesa and lib32-mesa PKGBUILD - Fixed 'IndexError: pop from empty list' error when building documentation with Sphinx 8.2.3 - Implemented solution based on issue sphinx-doc/sphinx#13407

chennin added the type:bug label Mar 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IndexError: pop from empty list when making mesa docs witn Sphinx 8.2 #13407

IndexError: pop from empty list when making mesa docs witn Sphinx 8.2 #13407

chennin commented Mar 3, 2025 •

edited

Loading

AA-Turner commented Mar 3, 2025

chennin commented Mar 3, 2025

AA-Turner commented Mar 3, 2025 •

edited

Loading

chennin commented Mar 3, 2025

AA-Turner commented Mar 3, 2025 •

edited

Loading

IndexError: pop from empty list when making mesa docs witn Sphinx 8.2 #13407

IndexError: pop from empty list when making mesa docs witn Sphinx 8.2 #13407

Comments

chennin commented Mar 3, 2025 • edited Loading

Describe the bug

How to Reproduce

Environment Information

Sphinx extensions

Additional context

AA-Turner commented Mar 3, 2025

chennin commented Mar 3, 2025

AA-Turner commented Mar 3, 2025 • edited Loading

chennin commented Mar 3, 2025

AA-Turner commented Mar 3, 2025 • edited Loading

chennin commented Mar 3, 2025 •

edited

Loading

AA-Turner commented Mar 3, 2025 •

edited

Loading

AA-Turner commented Mar 3, 2025 •

edited

Loading