Script to automate updating aspects of older OpenCPI Projects

General
1

I was recently updating an older OpenCPI project to the current style that ocpidev would generate, a lot of which is just moving and renaming various files.

So I wrote a script to help with the most common things I ended up doing:

https://pypi.org/project/ocpiupdate/

This will:

  • Move all Worker Description XML Files (OWDs) to use the newer <worker>-<model>.xml naming structure.
  • Move all specs/*[-_]spec.xml files to *.comp/*-comp.xml.
  • Inspect all OWD files for usage of these moved specs in the spec attribute and correct it appropriately.

I may add more to this over time as I need to do more updates, and I’ll comment if I do.

If you encounter any issues in using this, please ask for help.

Compatibility

I recommend Python 3.8 or newer (I wrote it for Ubuntu 20.04, and only used it on Python 3.8.10).

Usage

You can install this from PyPi:

pip install ocpiupdate

Or you can clone the repository, and build it with Hatch. This is currently left as an exercise for the reader, but I recommend using Astral’s uv.

You call it like this:

ocpiupdate --library <library> --project <project>

--library specifies a library to conduct the “Move” steps in. This can be specified more than once to operate over multiple libraries. The XML file / Makefile in this directory is ignored.

--project specifies a project to conduct the “Inspect” step in. This can be specified more than once to inspect multiple projects. This means that you can select a different project than the one you did the moves in, in case you have a second project that implements components from a first project. The XML file / Makefile in this directory is ignored.

Ignore any other flags at the moment.

Yes, I probably could have made these arguments more generic and useful, but this worked for me :slight_smile:

The script will print everything it does to the terminal, including:

  • The file moves
  • The directory creations
  • Which files it scans to do the find and replace
  • The exact string that was replaced, what it was replaced with, what line of the file it was from, and which file

Disclaimers

  • This is an unofficial script. I am not a core maintainer of OpenCPI.
  • This has only been run on one project.
    • Save your work to your version control system before attempting to use this.
  • The script has very little understanding of OpenCPI Project structure.
    • It only “parses” XML in very limited circumstances.

Known Issues

  • If this script renames the OWD of an RCC Worker that is a proxy of a HDL Worker, this will cause an error when attempting to build on v2.4.7 or earlier.
    • See this forum post.
    • Current workaround for this is to manually revert the name change that the script makes to these OWDs.

Potential Roadmap of future additions

  • Convert *_protocol.xml to *-prot.xml.
  • Have the script avoid the known issue documented above.
1 Like
2

Note: If you are using v2.4.7, this script is likely to cause you to encounter the issue documented in this forum post.

A fix for that issue has been merged into develop, so the next major release should avoid that issue.

For now, you will need to manually revert the *-rcc.xml change that the script makes for any RCC Workers that are proxies of HDL Workers.

Edit: Integrated this text into the original post.

3

Just updated this to hopefully avoid the RCC Worker proxy issue.

Also reorganised it to be easier to add stuff to in the future.

Haven’t tested it fully on this newer version, but hopefully it still works.

4

Small update: I’ve gone a little over the top. Here is the script in a repository rather than a snippet:

And here it is uploaded to PyPi:

That means you can now install it with:

pip install ocpiupdate

And call it as ocpiupdate in the Python environment you install it in.

5

I just pushed a v0.2.0 of this script to PYPI.

It fixes two bugs I found, and adds --dry-run and --version.

It also moves all debug messages to only print during --verbose.
However, messages saying that a file has moved or been edited will still print without --verbose.

6

Update

I just pushed v0.3.1 of ocpiupdate to PyPi.

Summary

This new version supports automatically migrating several of the “static” Makefiles in an OpenCPI Project. This includes:

  • application
  • hdl/adapters (Makefile and Library.mk)
  • hdl/assemblies
  • hdl/cards (Makefile and Library.mk)
  • hdl/devices (Makefile and Library.mk)
  • hdl/platforms
  • hdl/primitives
  • project (Makefile and Project.mk)

One of the difficulties when automating this conversion process, has been handling the various constructs that may be in a Makefile that ultimately don’t matter when converting it (checks that OCPI_CDK_DIR have been set, comments, etc).

This is handled by using the Python tree-sitter bindings, and examining each line of the parse for variable assignments. If anything else is found (that isn’t in the list of acceptable things to ignore), the update of that file is cancelled.

Limitations

  • Workers are not supported yet.
  • Updating existing XML files is not supported yet.
    • This obviously majorly impacts supporting Workers.

Configuration

The list of acceptable things to ignore is configurable via a config file; the default config file is here:

You can make a project local copy of this and use the --config flag to request that ocpiupdate use it.

This list is configurable per file type (the table ocpiupdate.parse.makefile) and per file (the table ocpiupdate.parse.makefile.<file-identifier>). Hopefully the documentation in this default config is enough for people other than me to use it, but ask for help if you need it.

This version also includes a second script called ocpiparsemake. This parses one Makefile and prints out its tree-sitter representation to the terminal. This is included to help when editing a config file in case you are unfamiliar with tree-sitter.

Finally, it is possible to add other static Makefile to XML conversions that I’ve missed by adding a new line to the ocpiupdate.migrate.makefile table, and then the associated ocpiupdate.parse.makefile.<file-identifier> and ocpiupdate.parse.xml.<file-identifier> tables as necessary.

7

Update

I just pushed v0.4.0 of ocpiupdate to PyPi.

Summary

This new version supports automatically migrating several of the “non-static” Makefiles in an OpenCPI Project. This includes:

  • components/*.hdl/Makefile
  • components/*.rcc/Makefile
  • components/*/*.hdl/Makefile
  • components/*/*.rcc/Makefile
  • hdl/*/*.hdl/Makefile (for the libraries adapters, cards, and devices)
  • hdl/*/*.rcc/Makefile (for the libraries adapters, cards, and devices)

One of the difficulties in moving over these Makefiles is the requirement to handle partially migrating to XML. That is where the XML file already exists and it needs to be added to.

Using any Python rich XML parser (xml.etree, xml.dom.minidom, lxml.etree, etc) would result in the XML files losing all of their whitespace formatting when they are saved. As such, I have once again used tree-sitter, but this time the XML bindings.

Python 3.11

This introduces a new issue: tree-sitter-xml is Python 3.9 and newer. This means I’d have to drop system python support for Ubuntu 20.04. Alongside this, there are several Python 3.11 features that I’d really like to use. Because of this:

ocpiupdate now requires >= Python 3.11.

This means that it will not run using the system python in Ubuntu 20.04 or Ubuntu 22.04.

Docker / Podman

To mitigate the impact of this, I have created and prebuilt a docker container that can be directly run over a project. See this section of the README for details on how to execute the docker images and the Gitlab Container Repository for the images themselves.

You can pull them down like this:

docker pull registry.gitlab.com/dawalters/ocpiupdate:v0.4.0

Alternative to Docker / Podman

As an alternative, you could use a python build tool to install a local python that doesn’t interfere with the system python. I recommend uv if you wish to do this instead.

Install it, then:

# This will download Python 3.11 if it can't find it locally
uv venv --python 3.11

source .venv/bin/activate

pip install ocpiupdate
1 Like