feat: add scyjava-stubgen cli command, and `scyjava.types` namespace, which provide type-safe imports with lazy init #82

tlambert03 · 2025-04-23T03:20:28Z

many more details and tests to follow... but just wanted to open this as a WIP.
edit: see #82 (comment) for details

Basic idea, after checking out this branch and running pip install -e . again:

create stubs with a cli commend, e.g. scyjava-stubgen org.scijava:parsington:3.1.0
Import names provided by that endpoint: python -c "from scyjava.types.org.scijava.parsington import Function; print(Function(1))". Only at the moment of class instantiation will the jvm be started.

- Introduced `scyjava-stubs` executable for generating Python type stubs from Java classes. - Implemented dynamic import logic in `_dynamic_import.py`. - Added stub generation logic in `_genstubs.py`. - Updated `pyproject.toml` to include new dependencies and scripts. - Created `__init__.py` for the `_stubs` package to expose key functionalities.

…mprove stub generation

…e in stubs

codecov · 2025-04-25T12:10:14Z

Codecov Report

Attention: Patch coverage is 77.46479% with 48 lines in your changes missing coverage. Please review.

Project coverage is 75.98%. Comparing base (34bfae2) to head (0d231cc).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
src/scyjava/_stubs/_cli.py	49.20%	32 Missing ⚠️
src/scyjava/_stubs/_genstubs.py	89.15%	9 Missing ⚠️
src/scyjava/_stubs/_dynamic_import.py	88.88%	4 Missing ⚠️
tests/test_stubgen.py	92.59%	2 Missing ⚠️
src/scyjava/_jvm.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #82       +/-   ##
===========================================
+ Coverage   52.72%   75.98%   +23.25%     
===========================================
  Files          12       20        +8     
  Lines        1303     1653      +350     
===========================================
+ Hits          687     1256      +569     
+ Misses        616      397      -219

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tlambert03 · 2025-05-01T00:22:29Z

hey @ctrueden, I'm struggling to run the jep tests locally (on my mac). I have openjdk version "11.0.27" 2025-04-15 but still get:

--> tests/it/java_heap.py [OK]
The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

Traceback (most recent call last):
  File "/Users/talley/dev/self/scyjava/tests/it/jvm_version.py", line 11, in <module>
    before_version = scyjava.jvm_version()
  File "/Users/talley/dev/self/scyjava/src/scyjava/_jvm.py", line 70, in jvm_version
    default_jvm_path = jpype.getDefaultJVMPath()
  File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 70, in getDefaultJVMPath
    return finder.get_jvm_path()
           ~~~~~~~~~~~~~~~~~~~^^
  File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 184, in get_jvm_path
    jvm = method()
  File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 311, in _javahome_binary
    return subprocess.check_output(
           ~~~~~~~~~~~~~~~~~~~~~~~^
        ['/usr/libexec/java_home']).strip()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/talley/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/subprocess.py", line 472, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               **kwargs).stdout
               ^^^^^^^^^
  File "/Users/talley/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/libexec/java_home']' returned non-zero exit status 1.

it's a bit unclear to me whether I should be trying to get these to work at all with jep, or just add a skip to the test

ctrueden · 2025-05-01T00:31:44Z

As a workaround, you could try setting JAVA_HOME? Then maybe jpype wouldn't try to invoke the java_home command. Out of curiosity: what does /usr/libexec/java_home -V say when you run it from the CLI?

tlambert03 · 2025-05-01T12:24:05Z

got that working... and commented out the "don't run on macos cause it's flaky" bit, and now get this:

-------------------------------------------
|  Testing Jep mode (Python inside Java)  |
-------------------------------------------
DEBUG 2025-05-01 08:20:23,154: Using settings:     {'m2repo': '/Users/talley/.m2/repository', 'cachedir': '/Users/talley/.jgo', 'links': 'auto'}
DEBUG 2025-05-01 08:20:23,155: Using repositories: {'scijava.public': 'https://maven.scijava.org/content/groups/public'}
DEBUG 2025-05-01 08:20:23,155: Using shortcuts:    {}
DEBUG 2025-05-01 08:20:23,155: Returning expanded coordinate black.ninia:jep:jep.Run.
DEBUG 2025-05-01 08:20:23,155: Returning expanded coordinate org.scijava:scijava-table.
INFO 2025-05-01 08:20:23,155: First time start-up may be slow. Downloaded dependencies will be cached for shorter start-up times in subsequent executions.
DEBUG 2025-05-01 08:20:23,155: Executing: ('/Users/talley/Library/Caches/cjdk/v0/misc-dirs/98cdba9371f93e1b5b8b95941b562d1647aecc21/apache-maven-3.9.9/bin/mvn', '-B', '-f', '/Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/pom.xml', 'dependency:resolve', '-X')
DEBUG 2025-05-01 08:20:31,600: Relevant mvn output: [INFO]    black.ninia:jep:jar:4.2.2:compile -- module jep (auto)
DEBUG 2025-05-01 08:20:31,600: Linking source /Users/talley/.m2/repository/black/ninia/jep/4.2.2/jep-4.2.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/jep-4.2.2.jar with link_type auto
DEBUG 2025-05-01 08:20:31,600: Linking source /Users/talley/.m2/repository/black/ninia/jep/4.2.2/jep-4.2.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/jep-4.2.2.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO]    org.scijava:scijava-table:jar:1.0.2:compile -- module org.scijava.table [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-table/1.0.2/scijava-table-1.0.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-table-1.0.2.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-table/1.0.2/scijava-table-1.0.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-table-1.0.2.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO]    org.scijava:scijava-common:jar:2.89.0:compile -- module org.scijava [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-common/2.89.0/scijava-common-2.89.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-common-2.89.0.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-common/2.89.0/scijava-common-2.89.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-common-2.89.0.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO]    org.scijava:parsington:jar:3.0.0:compile -- module org.scijava.parsington [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/parsington/3.0.0/parsington-3.0.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/parsington-3.0.0.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/parsington/3.0.0/parsington-3.0.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/parsington-3.0.0.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO]    org.bushe:eventbus:jar:1.4:compile -- module eventbus (auto)
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/bushe/eventbus/1.4/eventbus-1.4.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/eventbus-1.4.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/bushe/eventbus/1.4/eventbus-1.4.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/eventbus-1.4.jar with link_type hard
DEBUG 2025-05-01 08:20:31,602: Relevant mvn output: [INFO]    org.scijava:scijava-optional:jar:1.0.1:compile -- module org.scijava.optional [auto]
DEBUG 2025-05-01 08:20:31,602: Linking source /Users/talley/.m2/repository/org/scijava/scijava-optional/1.0.1/scijava-optional-1.0.1.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-optional-1.0.1.jar with link_type auto
DEBUG 2025-05-01 08:20:31,602: Linking source /Users/talley/.m2/repository/org/scijava/scijava-optional/1.0.1/scijava-optional-1.0.1.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-optional-1.0.1.jar with link_type hard
DEBUG 2025-05-01 08:20:31,602: class path: /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/*
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001716a3000 (most recent call first):
  <no Python frame>

that's probably not the flaky bit right? looks like a poor setup of the python environment

tlambert03 · 2025-05-01T12:30:54Z

i think i see the issue, i'll work on the jgo ... jep_test.py command directly and see if I can figure out what assumptions it's making about my environment setup

tlambert03 · 2025-05-01T13:10:35Z

after digging a bit deeper into how jgo and jep itself is working, I'm skipping stubgen tests on jep for now. it seems extremely dependent on some careful manual setup of the python environment. I was able to get it to find my standard library, but then it lost the pure python parts of jep. After fixing that, it was unable to find my (editable) install of scyjava because it's not following .pth files in a standard way. I don't understand all the variables well enough yet (I don't understand who's doing the magic, whether it's jep.Run itself or the jgo command line), so don't know where to attack it

tlambert03 · 2025-05-01T13:14:08Z

code-wise, this is ready to go... however, I don't expect it to be "human-interpretable" yet. There are many ways we could choose to document this and encourage/discourage its usage in various scenarios. Might be best to have a zoom about it so we can tinker with it together and discuss

imagesc-bot · 2025-05-19T20:32:55Z

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/fiji-friends-weekly-dev-update-thread/103718/94

imagesc-bot · 2025-06-03T21:11:51Z

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/poll-which-priorities-are-most-urgent-for-fijis-python-support/113172/1

ctrueden · 2025-08-15T17:49:34Z

Just a quick update: I removed the testing of jep mode awhile back, so that should no longer gum up the works here. @tlambert03 I'm hoping we can make a little time to revisit this PR some time next week.

tlambert03 · 2025-08-15T21:18:17Z

yes that would be great! i do think a little chat would be super useful here

tlambert03 · 2025-08-20T17:10:19Z

Writing some notes to self here (after having not looked at it for a while) that might help others understand what's going on here. This PR...

adds a new CLI command called scyjava-stubgen. It's worth mentioning this is the first CLI command offered by scyjava (i.e. this PR adds project.scripts to pyproject.toml for the first time). This command can be used to generate type stubs for any maven endpoint, for example: scyjava-stubgen org.scijava:parsington:3.1.0.
That CLI command just parses the arguments and calls the function scyjava._stubs.generate_stubs. (generate_stubs is a good candidate for a public function but I haven't touched the scyjava public API yet). Importantly, when called via the CLI it writes stubs by default to scyjava.types namespace itself. In other words it permantently writes stub files inside of the scyjava package (this could have permissions issues on some computers). For example, that parsington call above would create scyjava.types.org.scijava.parsington...
After running that command (and having generated stubs) you can now import those stubs directly from the namespace:
```
from scyjava.types.org.scijava.parsington import Function
```
with full static type support in IDEs. Which is great! that's the main goal here. 🎉
The important question to ask here, of course, is "when is the JVM started". That import alone does not start the JVM, instead, it returns a thin Proxy object that will start the JVM when an instance of that class is created (i.e. in the __new__ method). The logic for that is defined in scyjava._stubs.setup_java_imports. (That setup_java_imports function is used heavily by the generated type stubs, it generates the module level __getattr__ function that returns the Proxy objects imported from scyjava.types...)
an important detail here (as usual) is that all things would need to be imported before any of them are instantiated

open questions:

to me, the biggest open question here is "who is responsible for generating stubs, and where do they go?".

possibilities include:

after the environment and scyjava is setup. the user could call that command line argument. That's fine and good, but not "portable" since it requires action after pip install to get the type stubs.
Which brings the second part of this PR (which should probably be broken into a new PR), which are hatch and setuptools plugins to allow developers to build and include stubs in their own packages. More on that elsewhere

tlambert03 · 2025-08-20T18:39:18Z

after talking with @ctrueden ... something I will try to implement:

implement a sys.meta_path importer that sees imports from scyjava.types and dynamically generates types
implement a entry-point specification that allows packages to declare which endpoints they need (in order to generate dynamically imported types)

for this PR I will pull out everything by the stub generating mechanism, and do the above in another PR

tlambert03 · 2025-08-22T23:35:57Z

ok @ctrueden, I think this is ready for final review. I've removed the hatch/setuptools plugins, and double checked all the documentation. will follow up on usage patterns in another PR. Let me know if you have any questions!

tlambert03 added 15 commits April 22, 2025 22:50

ignore types folder

0f70b03

remove import

a8b2da7

add test

686739b

add comment to clarify stubgen command execution in test

7fe31af

refactor: clean up jpype imports in stubgen test and main module

afcc7a7

remove unused jpype import from _genstubs.py

a5cacc8

fix: add future annotations import to _cli.py

ab1bc2d

refactor: enhance dynamic_import function to accept base_prefix and i…

11649fa

…mprove stub generation

refactor: rename dynamic_import to setup_java_imports and update usag…

2cb4836

…e in stubs

reword

71f761e

feat: add Hatchling build hook for generating Java stubs

65cc471

wip

6e4181e

fix inclusion

6e92b13

add docs

0d231cc

tlambert03 added 3 commits April 25, 2025 08:17

setuptools plugin stub

79524b0

Merge branch 'main' into stubs

9bdba53

remove repr test

51937b5

skip in jep

192be35

remove setuptools plugin

e714abb

Merge branch 'main' into stubs

e7bc894

tlambert03 added 3 commits August 22, 2025 19:17

remove hatch plugin

a18d9d9

newlines

b53e795

update docs

ed0add6

tlambert03 mentioned this pull request Aug 23, 2025

scyjava.types metafinder #86

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add scyjava-stubgen cli command, and `scyjava.types` namespace, which provide type-safe imports with lazy init #82

feat: add scyjava-stubgen cli command, and `scyjava.types` namespace, which provide type-safe imports with lazy init #82

tlambert03 commented Apr 23, 2025 •

edited

Loading

Uh oh!

codecov bot commented Apr 25, 2025

Uh oh!

tlambert03 commented May 1, 2025 •

edited

Loading

Uh oh!

ctrueden commented May 1, 2025

Uh oh!

tlambert03 commented May 1, 2025

Uh oh!

tlambert03 commented May 1, 2025

Uh oh!

tlambert03 commented May 1, 2025

Uh oh!

tlambert03 commented May 1, 2025

Uh oh!

imagesc-bot commented May 19, 2025

Uh oh!

imagesc-bot commented Jun 3, 2025

Uh oh!

ctrueden commented Aug 15, 2025

Uh oh!

tlambert03 commented Aug 15, 2025

Uh oh!

tlambert03 commented Aug 20, 2025

Uh oh!

tlambert03 commented Aug 20, 2025

Uh oh!

tlambert03 commented Aug 22, 2025

Uh oh!

Uh oh!

feat: add scyjava-stubgen cli command, and scyjava.types namespace, which provide type-safe imports with lazy init #82

Are you sure you want to change the base?

feat: add scyjava-stubgen cli command, and scyjava.types namespace, which provide type-safe imports with lazy init #82

Conversation

tlambert03 commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 25, 2025

Codecov Report

Uh oh!

tlambert03 commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ctrueden commented May 1, 2025

Uh oh!

tlambert03 commented May 1, 2025

Uh oh!

tlambert03 commented May 1, 2025

Uh oh!

tlambert03 commented May 1, 2025

Uh oh!

tlambert03 commented May 1, 2025

Uh oh!

imagesc-bot commented May 19, 2025

Uh oh!

imagesc-bot commented Jun 3, 2025

Uh oh!

ctrueden commented Aug 15, 2025

Uh oh!

tlambert03 commented Aug 15, 2025

Uh oh!

tlambert03 commented Aug 20, 2025

open questions:

Uh oh!

tlambert03 commented Aug 20, 2025

Uh oh!

tlambert03 commented Aug 22, 2025

Uh oh!

Uh oh!

feat: add scyjava-stubgen cli command, and `scyjava.types` namespace, which provide type-safe imports with lazy init #82

feat: add scyjava-stubgen cli command, and `scyjava.types` namespace, which provide type-safe imports with lazy init #82

tlambert03 commented Apr 23, 2025 •

edited

Loading

tlambert03 commented May 1, 2025 •

edited

Loading