Document how to generate stubs for a branch or fork of MicroPython?

[samskiter]

Describe the bug
I've tried 2 or 3 times to work out how to use this to generate stubs for a custom micropython build for the RPi Pico W but I'm really struggling!

Here's what I've tried/gleaned so far:

• Some stubs can be generated on the board itself. I've managed to do this - I don't know if it's correct and I don't know where the firmware stubs are stored in the Stubs (https://github.com/Josverl/micropython-stubs) repo)
• There's some docstubs which can be generated from the micropython repository. These get stored in the stubs folder under some some directory called 'micropython--docstubs'
• There's some 'frozen' stubs - I don't know where these come from, I've managed to run get-frozen but don't know where the stubs that are 'got' are stored. I have tried deleting the stubs directory and rerunning get-frozen but despite apparent success, no folders seem to be created
• There's some 'core' stubs - get-core doesn't seem to create a micropython-core directory.
• There's merging stubs which produces a new directory with 'merged' in the name. I don't know how or whether this is used in later steps
• There's building the stubs - which uses the frozen and docstubs and maybe the generated stubs from the device. It presumably uses the merged directory? but it also seems to look for the frozen directory?
• There's a specific directory structure I need to check out. According to this I need to check out 4 repositories side-by-side and setup a symlink. But I think that's out of date, according to the docstring here in the github action: https://github.com/Josverl/micropython-stubs/blob/main/.github/workflows/get-all-frozen.yml

My guess for what I should do:

1. pip install micropython-stubber
2. checkout micropython-stubs
3. mkdir 'repos' inside micropython-stubs
4. checkout micropython & micropython-lib inside repos and checkout to the commit I want to generate stubs for.
5. Create a master branch at that commit
6. Run build of micropython for my pico w (do I need to do this?)
7. install micropython onto the pico w
8. Run create_stubs.py on the pico w (is this necessary?)
9. copy the generated stubs off the pico and place them into a folder inside micropython-stubs/stubs named micropython-latest-rp2-RPI_PICO_W
10. Run stubber get-frozen --black --version=latest
11. stubber get-docstubs
12. Run stubber merge --version latest --board RPI_PICO_W
13. Run stubber build --version latest --port rp2 --board RPI_PICO_W

This just isn't working for me - I'm getting failures about missing frozen stubs and all sorts - I must be doing something fundamentally wrong here or misunderstanding what's already checked in and what needs to be generated

[samskiter]

Ok, fixed a bug with using get-frozen with version=latest #442

[samskiter]

Update: I seem to be successfully using the get-frozen get-docstubs and merge commands now

• get-frozen reaches into the micropython repo and reads a bunch of manifest.py files to build up a list of pyi files. I had to add a fix to get this working here: Fix bug in get-frozen when using version=latest #442
• get-docstubs seems to reach into micropython/docs and 'compiles' a bunch of the RST files together and drops them into stubs/micropython--docstubs. There's no way to specify the board to my knowledge and it doesn't seem to work with version=latest, but ommitting version seems to default to latest... I'm unsure how the precedence works here - for example several 'default' values seem to appear in the output that don't seem to have a source (e.g. machine.adc init has a default value that I can't find specified anywhere). This process is a big mystical to me still - it's some kind of reverse application of sphinx?
• You can drop firmware stubs in stubs/micropython--- - e.g. micropython-latest-rp2-RPI_PICO_W in my case
• merge and get-docstubs will clean git up so if you're developing make sure you commit before running them
• my merge command looks like: stubber merge --version latest --port rp2 --board RPI_PICO_W
• build outputs into publish/micropython-latest-rp2-rpi_pico_w-stubs (note the lower case now). My build command looks like stubber build --version latest --port rp2 --board RPI_PICO_W - I haven't worked out how to get this output into VSCode yet - about to start trying

[samskiter]

My understanding of the workflow right now:

• Install micropython-stubber
• Clone micropython-stubs
• mkdir repos && cd repos
• clone micropython - checkout the desired point in history and force move the 'master' branch there
• clone micropython-lib - checkout the desired point in history and force move the 'master' branch there
• cd .. (aka move to the micropython-stubs repo) and start creating the stubs:
• CORE STUBS: run stubber get-core - This creates a bunch of folders like stubs/cpython_core-micropython
• FROZEN STUBS: run stubber get-frozen --black --version=latest - This should create stubs/micropython-latest-frozen
• DOCSTUBS: run stubber get-docstubs - this should create sstubs/micropython-latrest-docstubs
• FIRMWARE STUBS: build micropython and install it on yuour desired board & run create_stubs.py copy these firmware stubs stubs to stubs/micropython-latest-rp2-RPI_PICO_W - Instructions for this are elsewhere, but looks roughly like: cd repos/micropython (aka go into the micropython repo) git submodule update --init (aka get the submodules all checked out) make -C mpy-cross cd ports/rp2 make BOARD=RPI_PICO_W. Then you need to flash your version of micropython onto a board and run createstubs.py (I followed instructions here: https://github.com/paulober/Pico-W-Stub/blob/main/micropython-stubber.md#generating-the-stubs). Drag the files off an drop them in micropython-stubs/stubs/micropython-latest-rp2-RPI_PICO_W
• MERGED STUBS: run stubber merge --version latest --port rp2 --board RPI_PICO_W
• BUILD: run stubber build --version latest --port rp2 --board RPI_PICO_W

[samskiter]

The above doesn't work - seems that the frozen stubs overwrite the merged stubs - I thiunk I need to work out how to get the micropython repo to update the frozen bits

[Josverl]

Sam, thanks for your perseverance

This project started very small years ago - and has grown over time. documentation that I once wrote - has been surpassed by the reality of tools and micropython evolving.
also there is tech dept of some functionality - such as creating the firmware aka board stubs - that are not fully integrated into the packaged stubber tool, as I can't get them fully stable across all ports ( not esp8266)

writing this from memory in the train:

1. clone the micropython-stubber repo ( to get the scripts folder and the pyproject.toml file that holds the config)

2. either pip install micropython-stubber or poetry install

3. stubber clone --with stubs

4. optionally stubber clone --version v1.21.0
   This will clone the micropython, -lib and -stubs repos under /repos/*

5. run python scripts/board_stubber.py
   this will run the createstubs.py on the board , run post processing , merge them with the checked out stubs , and build them ( not sure about the build step)

6. the get-core command is obsoleted - it has been replaced by the get-docstubs which is actually better than get-core ever was. Il l need to deprecate that command.

[samskiter]

Sam, thanks for your perseverance

This project started very small years ago - and has grown over time. documentation that I once wrote - has been surpassed by the reality of tools and micropython evolving. also there is tech dept of some functionality - such as creating the firmware aka board stubs - that are not fully integrated into the packaged stubber tool, as I can't get them fully stable across all ports ( not esp8266)

writing this from memory in the train:

1. clone the micropython-stubber repo ( to get the scripts folder and the pyproject.toml file that holds the config)
2. either pip install micropython-stubber or poetry install
3. stubber clone --with stubs
4. optionally stubber clone --version v1.21.0
   This will clone the micropython, -lib and -stubs repos under /repos/*
5. run python scripts/board_stubber.py
   this will run the createstubs.py on the board , run post processing , merge them with the checked out stubs , and build them ( not sure about the build step)
6. the get-core command is obsoleted - it has been replaced by the get-docstubs which is actually better than get-core ever was. Il l need to deprecate that command.

Thanks for this and sorry for my previous impatience - felt so close to getting things working!

I think your instructions work but only for a specific tagged version. I'm testing a branch of micropython (micropython/micropython#10704) so I need to check out an arbitrary commit and generate stubs...

I notice you didn't mention using get-frozen or get-docstubs specifically but I think I will need those?

[Josverl]

I never considered a specific COMMIT, only versions to be honest.
You could add a local tag based on semver. I think something like v1.20.x could work.

Also if you do not specify a version, I think most commands just leave the repost on the commit they are. Or at least that is something that can be added with little change

[Josverl]

After looking at that PR I think I understand better what you are trying to do.
If there is a new DMA.rst file you'll indeed need to run get-docstubs and may need to run Get-frozen as well

Perhaps --version should also allow PR#1234 notation, let me think on that.
Still for this specific PR it will not be of any use,
as the PR does not add/change:

• any .rst file in docs/library
• any frozen .py file

comparing to main there are only 2 .rst files changed, however the files docs/rp2/general.rst and docs/rp2/quickref.rst are not used in generation stubs
so that means that you do not need to re-generate the docstubs , nor the frozen stubs.
You can recreate them , or re-use them from the nearest (lower) version

[samskiter]

--version=HEAD might be an interesting option and the produced output could be v<latest>-<sha> or v-adhoc-<sha> or something like that? - just some way to specify that I'd like it to be built right where we are...

Have you considered not checking in anything that is generated by the build? that might have made things clearer for me personally.

[Josverl]

you considered not checking in anything that is generated by the build?
Nothing is checked in automatically by the stubber

[Josverl]

just some way to specify that I'd like it to be built right where we are...

It already does that, if you just do not specify a version
Or do you have a specific command where you find this does not work?

All output that is not a specific tag, is considered 'latest', as that's exactly what I need, and keeps me from needing to do quite complex version resolution,

I see that if you are working on a year old PR that that is confusing, but it still works if you mentally time travel back to that PR.

It may be useful for your scenario if you remove all -latest- folders

Naming convention.
I do not think that using the Git hash for versioning is useful. It does not follow semver, nor is it human readable.
Also I do not see how it would be helpful in your scenario.

I may consider adding the pre-release build nr, now that that is adopted by micropython.
Still that would be 1.22.0-B1 and newer, and not solve parallel branches such as a PR or a fork

[samskiter]

It already does that, if you just do not specify a version Or do you have a specific command where you find this does not work?

Yea, I'm not sure that's working for me, OR I'm getting confused!
I think I couldn't find a way to get this to work for the merge and get-docstub commands because of this 'switch branch' command.

micropython-stubber/src/stubber/publish/candidates.py

Line 241 in 3779db1

r = git.switch_branch(repo=mpy_path, branch="master")

Latest also seems to be the default for those commands:

micropython-stubber/src/stubber/publish/candidates.py

Line 202 in 3779db1

versions: Union[str, List[str]] = V_LATEST,

It may be useful for your scenario if you remove all -latest- folders

That might be a good idea

Nothing is checked in automatically by the stubber

Sorry I meant the stubs library - are the outputs, of e.g. 'generate docstubs' not checked in there? Could these be viewed as intermediate-generate files?

I do not think that using the Git hash for versioning is useful. It does not follow semver, nor is it human readable.
Also I do not see how it would be helpful in your scenario.

Sorry I mean something like:
v1.19.0-a64d88
or
v-adhoc-a64d88

I have seen it as quite common practice to include a non-human-readable portion to version numbers (e.g. SHAs or build numbers) even for releases - this allows developer traceability back to a specific commit and build (as version numbers can be a little 'blurry' sometimes).

It would help my scenario of debug-devolpment because I could produce builds that I can produce stub builds that would let me easily identify where they came from...

It's been a week or so since I needed to touch this issue so some of this is from memory so sorry if this is a bit of a nonsensical reply :D