Neopo: A lightweight solution for local Particle development

[nrobinson2000]

Hello everyone.

I am proud to announce the release of neopo, a utility for locally building and flashing Particle projects to Particle devices that integrates with Particle Workbench.

TL;DR

I’ve made a replacement for po and po-util written in Python that is fully compatible with Particle Workbench and particle-cli. Here is the GitHub repository:

nrobinson2000/neopo

You can install neopo with the following command:

$ bash <( curl -sL https://git.io/JfwhJ )

Story

Four years ago I created po-util, a large, messy shell script for building, flashing, and interacting with Particle devices. After maintaining it for two years, it was in desperate need of a rewrite because the script was over 2,000 lines long.

I created a new repository called po and continued development there, breaking up the script into multiple files and taking on dependencies in a vain attempt to improve the maintainability of the project. Over the past year, I only made a few updates to po, letting the project grow stale. It grew stale because I had not fully integrated the project with newly developed tools like Workbench and the improvements made to particle-cli.

Since the initial release of Workbench, I dreamed of integrating its features and dependencies into a slick, useful tool that can be used on Linux and macOS independent from the GUI of Visual Studio Code.

In the past week, I’ve realized that dream. Neopo natively uses the Workbench project format and relies on the same dependencies and Makefile. It can even manage dependencies as well. Overall, not only is neopo an excellent companion to Workbench, but it offers unparalleled utility in build environments where Workbench is not an option as well.

To all users of po, I highly recommend switching over to neopo for your local Particle needs, as it is an immense improvement over the previous project. To everyone else, I would greatly appreciate it if you could give neopo a try.

I await your feedback and bug reports. Feel free to make an issue on the GitHub repository.

nrobinson2000/neopo

[nrobinson2000]

Update: Raspberry Pi support

Last night I compiled gcc-arm 5.3.1 for the Raspberry Pi. I have uploaded the compiled toolchain to the neopo repository on GitHub. Now when you install neopo on a Raspberry Pi it will download my compiled toolchain instead of the x86 toolchain hosted on binaries.particle.io

I’ve also fixed the install script to download the required dependencies for Particle CLI.

Neopo now works fully on Raspberry Pi.

---

I timed the compilation of an empty cleaned project on a Raspberry Pi4 with:

time neopo run compile-all

It took 3m53.841s to compile, compared to my Linux desktop which took 0m45.049s.

Building again without cleaning first took 0m7.179s on the Pi4, versus 0m1.622s on my Linux desktop.

[nrobinson2000]

If anyone wants to try to get neopo working on Windows I’d greatly appreciate it if you could share your results.

Windows testing/support: Help wanted

[gusgonnet]

great work Nathan.

Why would one use neopo vs just using the Workbench?
I looked at the examples and couldn’t figure it out.
Thanks!

[nrobinson2000]

Thanks Gustavo.

Since the early days of Particle Dev and Particle Workbench, there have been many scenarios where other developers and I have needed a tool that could be used to quickly compile a project or flash firmware to a device without having the time or interest to open up a large IDE.

Back when the Web IDE was king and before Workbench was released, I attempted to build a wrapper around the makefiles in deviceOS, called po-util, or later po, that provided that experience, but it was large and clumsy because the neatly organized and now easily available dependencies of Workbench had not been released yet.

Neopo is most useful for developers who are confident with the command line, as it provides a powerful CLI experience much like Particle CLI but it offers the build features of Workbench.

It’s also incredibly portable and extensible, meaning that it can be easily used in automated, restricted, or remote environments like Travis CI, Raspberry Pi, or any macOS or Linux (Ubuntu/Debian) host.

Since the core functionality of neopo is kept in a single Python script, savvy users can integrate the module into existing Python solutions, providing possibilities for sophisticated Particle development workflows. I’ve done something similar in the past, integrating po into a script that could be used to flash firmware and import devices into a product in series, greatly accelerating the provisioning process.

While neopo is complementary to Particle Workbench, it is by no means a replacement. I have no idea how to implement Intellisense or the Debugger on the command line or in an editor like Vim. Overall, I hope that with the redesigned tool I can provide an improved experience for developers who had used po or po-util in the past, and reach new users who may be interested in improving their Particle development workflow.

[gusgonnet]

alright, thanks Nathan for the extra explanation!
Gustavo.

[nrobinson2000]

Update: Windows support

I have added Windows (Cygwin) support for neopo. Here are the installation instructions: https://github.com/nrobinson2000/neopo/blob/master/docs/windows.md

[nrobinson2000]

Update:

I have made tons of improvements to neopo over the last month. I’ve also written some documentation: https://nrobinson2000.github.io/neopo/full-docs.html

[nrobinson2000]

Update: New domain

I purchased neopo.xyz today to host the documentation and install script.

Neopo can now also be installed using the following:

bash <(curl -sL neopo.xyz/install)

[rvnash]

Nice work and thank you, Nathan.

I’ve been using neopo for a few days now in my projects and it’s working great. The reason why I’ve always wanted something like this is because I want to be able to do command-line local compiles and deployment. This is really the first practical easy to use way to do this.

I will also say that the exact same project which works well with neopo, will also open and build great with Particle Workbench, and can compile using the Particle CLI.

Also note that the compiles that come out of neopo are exactly the same (byte size) as those that come from Particle Workbench. That gives me some reassurance that neopo is compatible.

[nrobinson2000]

Update: Support for deviceOS 2.0.0 and gcc-arm 9.2.1

I have made improvements to neopo to select the appropriate gcc-arm version when compiling a project and download gcc-arm versions on the fly if needed.

I compiled gcc-arm 9.2.1 natively on Raspberry Pi 4 so that projects using deviceOS 2.0.0 can be built successfully.

To upgrade neopo to get these new features simply run:

$ neopo upgrade

[nrobinson2000]

Update: Support for building on aarch64

After many hours of compilation and needing to get a larger micro SD card I have produced gcc-arm 5.3.1 and 9.2.1 toolchains for aarch64.

I used Manjaro ARM XFCE on Raspberry Pi 4 to produce the toolchain and test neopo.

I have not updated the universal install script to detect aarch64 yet, but for anyone who is using Manjaro or Arch on Raspberry Pi or other aarch64 systems the following commands are the required steps to install neopo. Afterwards you can use neopo like on any other Linux environment.

# Install essential dependencies if you haven't already
sudo pacman -S base-devel

# Clone neopo and prepare to create package
git clone https://github.com/nrobinson2000/neopo
cd aur/

# Edit PKGBUILD to comment out x86_64 depends and uncomment aarch64 depends
vim PKGBUILD

# Create and install the neopo-git package
makepkg -si

# Install neopo dependencies
neopo install

# Fix and install particle-cli
export GODE_DEBUG=info  # This is optional
neopo particle # This will fail

# Replace node executable
mv ~/.particle/node-v12*/bin/node ~/.particle/node-v12*/bin/node.old
ln -s $(which node) ~/.particle/node-v12*/bin/node

# Complete particle-cli install and setup non-root USB access
neopo particle configure usb

UPDATE:

I have started my own package repository on GitHub for x86_64 and aarch64 packages. You can install neopo with the following on Manjaro/Arch:

# Install essential dependencies if you haven't already
sudo pacman -S base-devel

# Clone the package repository and install neopo
git clone https://github.com/nrobinson2000/packages
cd packages
./install-pkg neopo

All of the commands I listed above to fix the installation are run automatically after building and installing the package.

UPDATE:

I have modified the universal installer to use nrobinson2000/packages to install neopo. Now on any x86_64 or aarch64 system running Manjaro/Arch you can use the universal installer to properly install neopo as a package.

bash <(curl -sL neopo.xyz/install)

[nrobinson2000]

Update: neopo-git on the AUR

I have submitted neopo-git to the Arch User Repository.

yay -G neopo-git
cd neopo-git && makepkg -sif

Due to a bug in yay, the commands above are required to install neopo on aarch64.

On x86_64 the following will suffice:

yay -S neopo-git

Until the bug in yay is resolved, I recommend using the universal installer as it will work on either architecture and apply post-installation tweaks automatically.

[nrobinson2000]

Update: Manpage for neopo, better Python support

I have finally written a manpage for neopo. It can be accessed with:

$ man neopo

I’ve made an online version available: https://neopo.xyz/docs/neopo.html

Additionally, when installing neopo it is now properly built as a Python module using setuptools/pip, providing better support for using neopo within scripts and on the command line. In future releases, I plan to break the neopo source into separate files and follow Python conventions better.

I’m in the process of migrating from Travis CI to GitHub Actions. You can see the GitHub actions builds for neopo here.

[nrobinson2000]

UPDATE: Workbench on Pi 400 and other improvements

---

TL;DR? I’ve managed to get Particle Workbench running on my Raspberry Pi 400 by leveraging a few of the new features I’ve developed for neopo. And yes, that is the Particle debugger running.

Enhancements

• Added the neopo setup command which applies necessary tweaks to complete the installation of Particle toolchains on Manjaro x86_64 and aarch64.
• Added the neopo setup-workbench command which installs the Workbench extension pack into an existing VSCode installation and applies several tweaks to make Workbench usable on aarch64.
• Added the neopo-script command which is separate program installed in the path that functions identically to neopo script but without the overhead of importing the rest of the package. Additionally scripts can now be passed into neopo script from standard input.

Changes

• Divided neopo.py into separate files, cleaning up the codebase.
• Improved the scripting interface and the Python module API.
• Updated setup.py to provide particle, so users can easily access an encapsulated Particle CLI instance using a temporary environment that eliminates many common issues.
• Removed the neopo load command as enforcing a centralized script directory was counterproductive.

---

Installing neopo:

The following will work with most Linux distributions and macOS:

$ bash <(curl -sL neopo.xyz/install)

On Manjaro/Arch the universal installer listed above essentially does the following:

$ pacman -S --needed yay base-devel
$ yay -G neopo-git
$ cd neopo-git
$ makepkg -sif
$ neopo setup

If you are interested in using Workbench on Manjaro ARM (like I am on my Raspberry Pi 400) you can run the following to first install VSCode (if you haven’t already) and then install the Workbench extensions:

$ yay -S visual-studio-code-bin
$ neopo setup-workbench

[nrobinson2000]

UPDATE: Improved dependency management and support for custom Device OS

---

Enhancements

• Added the neopo remove command which can delete a specified installed Device OS release.
• Added support for the NEOPO_LOCAL and NEOPO_PATH environment variables which can let neopo store dependencies in locations other than ~/.particle/toolchains.
• Added support for custom Device OS installations. (Tutorial below.)
• Improved the neopo versions command to list custom Device OS versions and platforms that have been manually installed.

Changes

• Various internal changes to how dependency management is performed.

---

Installing and building with custom deviceOS

Download and setup 3.0.0-beta.1:

$ cd ~/.particle/toolchains/deviceOS
$ git clone https://github.com/particle-iot/device-os 3.0.0-beta.1
$ cd 3.0.0-beta.1
$ git submodule update --init --recursive

Configure and build a project:

$ neopo configure argon 3.0.0-beta.1 myProject
$ neopo build myProject

---

Download and setup redbearduo:

$ cd ~/.particle/toolchains/deviceOS
$ git clone https://github.com/redbear/firmware redbearduo
$ cd redbearduo
$ git submodule update --init --recursive
$ mkdir .workbench
$ curl -sLo .workbench/manifest.json https://git.io/JLyYI

Configure and build a project:

$ neopo configure duo redbearduo myProject
$ neopo build myProject

[nrobinson2000]

UPDATE: manifest.json hotfix

In the latest version of the Workbench extension, manifest.json was moved from extension/src/compiler/manifest.json to extension/node_modules/@particle/toolchain-manager/manifest.json, breaking existing installations of neopo.

I have made the necessary changes to prevent the following error from occurring:

$ neopo install
Installing neopo...
Finding Workbench extension URL...
Downloading Workbench extension...
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/neopo/command.py", line 211, in main
    commands[args[1]](args)
  File "/usr/lib/python3.9/site-packages/neopo/command.py", line 26, in install_command
    install_or_update(True, force)
  File "/usr/lib/python3.9/site-packages/neopo/workbench.py", line 167, in install_or_update
    data = get_deps()
  File "/usr/lib/python3.9/site-packages/neopo/workbench.py", line 126, in get_deps
    manifest = get_file(extension, extensionFiles["manifest"])
  File "/usr/lib/python3.9/site-packages/neopo/workbench.py", line 59, in get_file
    return file.read(path)
  File "/usr/lib/python3.9/zipfile.py", line 1463, in read
    with self.open(name, "r", pwd) as fp:
  File "/usr/lib/python3.9/zipfile.py", line 1502, in open
    zinfo = self.getinfo(name)
  File "/usr/lib/python3.9/zipfile.py", line 1429, in getinfo
    raise KeyError(
KeyError: "There is no item named 'extension/src/compiler/manifest.json' in the archive"
An unexpected error occurred!
To report this error on GitHub, please open an issue:
https://github.com/nrobinson2000/neopo/issues

To avoid any interruption, please update neopo to the latest version.

On Linux and macOS:

$ bash <(curl -sL neopo.xyz/install)

From the AUR:

$ yay -S neopo-git
$ neopo install

From source or on Windows:

$ git clone https://github.com/nrobinson2000/neopo
$ cd neopo
$ python3 -m pip install .
$ neopo install

[nrobinson2000]

UPDATE: Docker image

As mentioned in this thread I have created a docker image for neopo.

The image is currently 181MB when pulled, and 1.14GB after installing Workbench dependencies.

Pulling and running the container:

$ docker pull nrobinson2000/neopo
$ docker run -it nrobinson2000/neopo

Example usage of neopo inside container:

# neopo install
# particle --version
# neopo create myProject
# neopo configure boron 2.0.1 myProject
# neopo build myProject -v

---

Here is a link to a recorded demo:

[gusgonnet]

you are the man, thanks Nathan

[nrobinson2000]

UPDATE: Bash completion improvements

I have made several improvements to both the neopo completion and particle completion scripts I maintain. These scripts provide incredible value when using particle and neopo.

I have updated the AUR package to additionally include the particle completion script along with the neopo completion script.

Both scripts are now enabled when using the neopo docker image. Additionally, the neopo dependencies are now pre-installed in the image. The image is now 175MB.

Here is a demo showing the scripts in action:
https://asciinema.org/a/392701