Neopo: A lightweight solution for local Particle development

Nice work and thank you, Nathan.

I’ve been using neopo for a few days now in my projects and it’s working great. The reason why I’ve always wanted something like this is because I want to be able to do command-line local compiles and deployment. This is really the first practical easy to use way to do this.

I will also say that the exact same project which works well with neopo, will also open and build great with Particle Workbench, and can compile using the Particle CLI.

Also note that the compiles that come out of neopo are exactly the same (byte size) as those that come from Particle Workbench. That gives me some reassurance that neopo is compatible.

1 Like

Update: Support for deviceOS 2.0.0 and gcc-arm 9.2.1

I have made improvements to neopo to select the appropriate gcc-arm version when compiling a project and download gcc-arm versions on the fly if needed.

I compiled gcc-arm 9.2.1 natively on Raspberry Pi 4 so that projects using deviceOS 2.0.0 can be built successfully.

To upgrade neopo to get these new features simply run:

$ neopo upgrade
1 Like

Update: Support for building on aarch64

After many hours of compilation and needing to get a larger micro SD card I have produced gcc-arm 5.3.1 and 9.2.1 toolchains for aarch64.

I used Manjaro ARM XFCE on Raspberry Pi 4 to produce the toolchain and test neopo.

I have not updated the universal install script to detect aarch64 yet, but for anyone who is using Manjaro or Arch on Raspberry Pi or other aarch64 systems the following commands are the required steps to install neopo. Afterwards you can use neopo like on any other Linux environment.

# Install essential dependencies if you haven't already
sudo pacman -S base-devel

# Clone neopo and prepare to create package
git clone https://github.com/nrobinson2000/neopo
cd aur/

# Edit PKGBUILD to comment out x86_64 depends and uncomment aarch64 depends
vim PKGBUILD

# Create and install the neopo-git package
makepkg -si

# Install neopo dependencies
neopo install

# Fix and install particle-cli
export GODE_DEBUG=info  # This is optional
neopo particle # This will fail

# Replace node executable
mv ~/.particle/node-v12*/bin/node ~/.particle/node-v12*/bin/node.old
ln -s $(which node) ~/.particle/node-v12*/bin/node

# Complete particle-cli install and setup non-root USB access
neopo particle configure usb

UPDATE:

I have started my own package repository on GitHub for x86_64 and aarch64 packages. You can install neopo with the following on Manjaro/Arch:

# Install essential dependencies if you haven't already
sudo pacman -S base-devel

# Clone the package repository and install neopo
git clone https://github.com/nrobinson2000/packages
cd packages
./install-pkg neopo

All of the commands I listed above to fix the installation are run automatically after building and installing the package.

UPDATE:

I have modified the universal installer to use nrobinson2000/packages to install neopo. Now on any x86_64 or aarch64 system running Manjaro/Arch you can use the universal installer to properly install neopo as a package.

bash <(curl -sL neopo.xyz/install)
2 Likes

Update: neopo-git on the AUR

I have submitted neopo-git to the Arch User Repository.

yay -G neopo-git
cd neopo-git && makepkg -sif

Due to a bug in yay, the commands above are required to install neopo on aarch64.

On x86_64 the following will suffice:

yay -S neopo-git

Until the bug in yay is resolved, I recommend using the universal installer as it will work on either architecture and apply post-installation tweaks automatically.

1 Like

Update: Manpage for neopo, better Python support

I have finally written a manpage for neopo. It can be accessed with:

$ man neopo

I’ve made an online version available: https://neopo.xyz/docs/neopo.html

Additionally, when installing neopo it is now properly built as a Python module using setuptools/pip, providing better support for using neopo within scripts and on the command line. In future releases, I plan to break the neopo source into separate files and follow Python conventions better.

I’m in the process of migrating from Travis CI to GitHub Actions. You can see the GitHub actions builds for neopo here.

2 Likes

UPDATE: Workbench on Pi 400 and other improvements


TL;DR? I’ve managed to get Particle Workbench running on my Raspberry Pi 400 by leveraging a few of the new features I’ve developed for neopo. And yes, that is the Particle debugger running.

Enhancements

  • Added the neopo setup command which applies necessary tweaks to complete the installation of Particle toolchains on Manjaro x86_64 and aarch64.
  • Added the neopo setup-workbench command which installs the Workbench extension pack into an existing VSCode installation and applies several tweaks to make Workbench usable on aarch64.
  • Added the neopo-script command which is separate program installed in the path that functions identically to neopo script but without the overhead of importing the rest of the package. Additionally scripts can now be passed into neopo script from standard input.

Changes

  • Divided neopo.py into separate files, cleaning up the codebase.
  • Improved the scripting interface and the Python module API.
  • Updated setup.py to provide particle, so users can easily access an encapsulated Particle CLI instance using a temporary environment that eliminates many common issues.
  • Removed the neopo load command as enforcing a centralized script directory was counterproductive.

Installing neopo:

The following will work with most Linux distributions and macOS:

$ bash <(curl -sL neopo.xyz/install)

On Manjaro/Arch the universal installer listed above essentially does the following:

$ pacman -S --needed yay base-devel
$ yay -G neopo-git
$ cd neopo-git
$ makepkg -sif
$ neopo setup

If you are interested in using Workbench on Manjaro ARM (like I am on my Raspberry Pi 400) you can run the following to first install VSCode (if you haven’t already) and then install the Workbench extensions:

$ yay -S visual-studio-code-bin
$ neopo setup-workbench
2 Likes

UPDATE: Improved dependency management and support for custom Device OS


Enhancements

  • Added the neopo remove command which can delete a specified installed Device OS release.
  • Added support for the NEOPO_LOCAL and NEOPO_PATH environment variables which can let neopo store dependencies in locations other than ~/.particle/toolchains.
  • Added support for custom Device OS installations. (Tutorial below.)
  • Improved the neopo versions command to list custom Device OS versions and platforms that have been manually installed.

Changes

  • Various internal changes to how dependency management is performed.

Installing and building with custom deviceOS

Download and setup 3.0.0-beta.1:

$ cd ~/.particle/toolchains/deviceOS
$ git clone https://github.com/particle-iot/device-os 3.0.0-beta.1
$ cd 3.0.0-beta.1
$ git submodule update --init --recursive

Configure and build a project:

$ neopo configure argon 3.0.0-beta.1 myProject
$ neopo build myProject

Download and setup redbearduo:

$ cd ~/.particle/toolchains/deviceOS
$ git clone https://github.com/redbear/firmware redbearduo
$ cd redbearduo
$ git submodule update --init --recursive
$ mkdir .workbench
$ curl -sLo .workbench/manifest.json https://git.io/JLyYI

Configure and build a project:

$ neopo configure duo redbearduo myProject
$ neopo build myProject

UPDATE: manifest.json hotfix

In the latest version of the Workbench extension, manifest.json was moved from extension/src/compiler/manifest.json to extension/node_modules/@particle/toolchain-manager/manifest.json, breaking existing installations of neopo.

I have made the necessary changes to prevent the following error from occurring:

$ neopo install
Installing neopo...
Finding Workbench extension URL...
Downloading Workbench extension...
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/neopo/command.py", line 211, in main
    commands[args[1]](args)
  File "/usr/lib/python3.9/site-packages/neopo/command.py", line 26, in install_command
    install_or_update(True, force)
  File "/usr/lib/python3.9/site-packages/neopo/workbench.py", line 167, in install_or_update
    data = get_deps()
  File "/usr/lib/python3.9/site-packages/neopo/workbench.py", line 126, in get_deps
    manifest = get_file(extension, extensionFiles["manifest"])
  File "/usr/lib/python3.9/site-packages/neopo/workbench.py", line 59, in get_file
    return file.read(path)
  File "/usr/lib/python3.9/zipfile.py", line 1463, in read
    with self.open(name, "r", pwd) as fp:
  File "/usr/lib/python3.9/zipfile.py", line 1502, in open
    zinfo = self.getinfo(name)
  File "/usr/lib/python3.9/zipfile.py", line 1429, in getinfo
    raise KeyError(
KeyError: "There is no item named 'extension/src/compiler/manifest.json' in the archive"
An unexpected error occurred!
To report this error on GitHub, please open an issue:
https://github.com/nrobinson2000/neopo/issues

To avoid any interruption, please update neopo to the latest version.

On Linux and macOS:

$ bash <(curl -sL neopo.xyz/install)

From the AUR:

$ yay -S neopo-git
$ neopo install

From source or on Windows:

$ git clone https://github.com/nrobinson2000/neopo
$ cd neopo
$ python3 -m pip install .
$ neopo install

UPDATE: Docker image

As mentioned in this thread I have created a docker image for neopo.

The image is currently 181MB when pulled, and 1.14GB after installing Workbench dependencies.

Pulling and running the container:

$ docker pull nrobinson2000/neopo
$ docker run -it nrobinson2000/neopo

Example usage of neopo inside container:

# neopo install
# particle --version
# neopo create myProject
# neopo configure boron 2.0.1 myProject
# neopo build myProject -v

Here is a link to a recorded demo:

asciicast

1 Like

you are the man, thanks Nathan :fist_left:

UPDATE: Bash completion improvements

I have made several improvements to both the neopo completion and particle completion scripts I maintain. These scripts provide incredible value when using particle and neopo.

I have updated the AUR package to additionally include the particle completion script along with the neopo completion script.

Both scripts are now enabled when using the neopo docker image. Additionally, the neopo dependencies are now pre-installed in the image. The image is now 175MB.

Here is a demo showing the scripts in action:
https://asciinema.org/a/392701

Thank you for this tremendous work, Nathan. I was looking for something to help me automate builds with my gitlab CI/CD. I have not plumbed it out all the way yet, but I did some testing with the docker image and it looks like this will make it work.

4 Likes

I would be happy to help out with testing and documentation of the installation. However, I cannot figure out from what is published, how to install it for Windows. I already have Python and Git installed (I use them).

Here is where I get stuck:

$ python3 -m pip install .
$ neopo install

“python3 -m pip install .” should reference a neopo installation in some way, but it doesn’t.

Hey @markwkiehl, sorry for the confusion.

You need to run the pip install command from within the cloned neopo repository.

The following should work in your case:

git clone https://github.com/nrobinson2000/neopo
cd neopo
python3 -m pip install .
neopo install

Raspberry Pi manifest.json bug is back

Hi Nathan. I’ve been using neopo and it has helped tremendously with productivity. I hope you are still working the project.

I experienced the “manifest.json bug” of 2021 and it seems to have returned on the RPi. It throws the missing manifest error on installation and thereafter cannot download the executables.

I have tried it on 2 x86 Ubuntu boxes and it installs fine, so it seems to be specific to RPi.

These are RPi4 with the latest (last week’s) full “Raspberry Pi OS with desktop and common applications” installed.

I have tried with Particle pre-installed, and without, with same results. A typical log is pasted below.

Any help you can provide will be most welcome.

Thanks,
Bill

pi@raspberrypi:~ $ particle version
3.7.0
pi@raspberrypi:~ $ which neopo
pi@raspberrypi:~ $ bash <(curl -sL neopo.xyz/install)
Installing Linux dependencies...
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
dfu-util is already the newest version (0.9-1).
git is already the newest version (1:2.30.2-1).
jq is already the newest version (1.6-2.1).
libarchive-zip-perl is already the newest version (1.68-1).
libfl-dev is already the newest version (2.6.4-8).
libisl15 is already the newest version (0.18-4).
libncurses5 is already the newest version (6.2+20201114-2).
libudev-dev is already the newest version (247.3-7+rpi1+deb11u1).
libusb-1.0-0-dev is already the newest version (2:1.0.24-3).
python3 is already the newest version (3.9.2-3).
python3-setuptools is already the newest version (52.0.0-4).
vim is already the newest version (2:8.2.2434-3+deb11u1).
python3-pip is already the newest version (20.3.4-4+rpt1+deb11u1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Cloning into '/tmp/tmp.fstOq0UvT6/neopo'...
remote: Enumerating objects: 1785, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 1785 (delta 4), reused 7 (delta 4), pack-reused 1768
Receiving objects: 100% (1785/1785), 3.32 MiB | 1.41 MiB/s, done.
Resolving deltas: 100% (1040/1040), done.
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing /tmp/tmp.fstOq0UvT6/neopo
Building wheels for collected packages: neopo
  Building wheel for neopo (setup.py) ... done
  Created wheel for neopo: filename=neopo-361-py3-none-any.whl size=39744 sha256=8c3ea1e73b4437c0ea1ec6d3516de547d54787d657d8a4a6178ee2e7b2716c12
  Stored in directory: /tmp/pip-ephem-wheel-cache-0tb3r9ty/wheels/38/70/a2/beb426c12a78cb4bd87e76038794153f054ca7845f71c7dde2
Successfully built neopo
Installing collected packages: neopo
Successfully installed neopo-361
Installing neopo...
Finding Workbench extension URL...
Downloading Workbench extension...
Failed to download particle executable from extension!
Error: file /home/pi/.neopo/cache/manifest.json not found.
Please ensure that you have installed the dependencies:
	$ neopo install
pi@raspberrypi:~ $ neopo update .
Updating dependencies...
Finding Workbench extension URL...
Downloading Workbench extension...
Failed to download particle executable from extension!
pi@raspberrypi:~ $ neopo upgrade
This command is deprecated because neopo is now installed using pip or the AUR.
To upgrade neopo, either rerun the universal installer, or follow distribution
specific instructions.

Hey @wawhite3

The root cause of this error is that the Particle Workbench extension no longer ships with a particle binary for ARM. However, it appears that the binary is still available from binaries.particle.io

I’ve released a fix to neopo to correct this error. Please try updating/reinstalling neopo.

1 Like

Thanks Nathan, I will let you know if I have any further troubles.

Hi Nathan, I was having issues installing Particle on my Raspberry Pi when I came across your service which seems sorely needed. Thank you for maintaining this project throughout the years. The issues I was having was when trying to run “particle setup” it would result in the error:
â Now to find your device(s)…TypeError: SerialPort.list is not a function

From what I could tell it seems neopo has the fix to this issue. However, I am now running into the same problems as wawhite3 above. All of the same commands yield the same errors they got. I am running 2 raspberry pi zeros with the current light raspbian OS with the desktop GUI installed and both of them run into the same issues.
“neopo install” returns:
“Downloading Workbench extension…
Failed to download particle executable from extension!”

“neopo particle login” or “neopo login setup” returns:
$Error: file /home/pi/.neopo/cache/manifest.json not found.
$Please ensure that you have installed the dependencies:
$ neopo install

Any help you can give me would be greatly appreciated.
Thanks for your time,
Harry

Hey @carburyharry

Thanks for using neopo!

Two questions:

  • Are you using the latest neopo commit with the fix for the missing Particle executable?
  • Are you using 32-bit or 64-bit Raspberry Pi OS? (Both architectures are supported, but aarch64 requires some additional post-install setup.)

I would recommend trying to manually install neopo:

git clone https://github.com/nrobinson2000/neopo
cd neopo
sudo python3 -m pip install .
neopo install

Also, it’s unlikely, but it might be possible that you have multiple (conflicting) installations of neopo. You can check with:

which -a neopo

I rebuilt a fresh armv7h Arch Linux ARM install for Raspberry Pi 4 last night and tested the latest neopo-git, and everything still seems to be working.

I’ve been using Arch Linux ARM on all of my Raspberry Pi’s lately, and it’s far less of a headache than stock Raspberry Pi OS. I’ve made a tool to prepare configured bootable drives for Arch Linux ARM, but it requires an x86_64 host running Arch Linux to run. I’ll publish the tool soon.

Hey @nrobinson2000,
Thanks for getting back to me, to answer your first question, yes I believe I was using the latest neopo commit as I attempted to install using the Universal Installer and install from source methods listed on your website https://neopo.xyz/. The source method required the installation of python 3 beforehand as raspbian lite does not have it.
Secondly, I am using Raspbian OS Lite 32-bit installed using the raspberry pi imager, I’m still very new to this but I didn’t know if the pi zero w could run 64-bit.

So I attempted the methods you gave me, on both Pi’s when I run

which -a neopo

it returns

 /usr/local/bin/neopo

and when manually installing neopo as you outlined above…

carburyharry@raspberrypi:~ $ git clone https://github.com/nrobinson2000/neopo
Cloning into 'neopo'...
remote: Enumerating objects: 1789, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 1789 (delta 7), reused 12 (delta 5), pack-reused 1768
Receiving objects: 100% (1789/1789), 3.32 MiB | 669.00 KiB/s, done.
Resolving deltas: 100% (1043/1043), done.
carburyharry@raspberrypi:~ $ cd neopo
carburyharry@raspberrypi:~/neopo $ ^[[200~sudo python3 -m pip install .~
bash: $'\E[200~sudo': command not found
carburyharry@raspberrypi:~/neopo $ sudo python3 -m pip install .
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing /home/carburyharry/neopo
Building wheels for collected packages: neopo
  Building wheel for neopo (setup.py) ... done
  Created wheel for neopo: filename=neopo-362-py3-none-any.whl size=39977 sha256=2b95a17f90a63674077a4a4a70ea36caa19f1fb43bbd0b6d809ffa775c5a2e85
  Stored in directory: /tmp/pip-ephem-wheel-cache-cvhioulm/wheels/c7/03/ff/9dfe1ee4abcd34cac8fd9d533e090da8e7d89629e2b77e87d7
Successfully built neopo
Installing collected packages: neopo
  Attempting uninstall: neopo
    Found existing installation: neopo 362
    Uninstalling neopo-362:
      Successfully uninstalled neopo-362
Successfully installed neopo-362
carburyharry@raspberrypi:~/neopo $ neopo install
Installing neopo...
Finding Workbench extension URL...
Downloading Workbench extension...
Failed to download particle executable from extension!

Same result on both Pi’s.
So unfortunately still no luck, I do also have access to a raspberry pi 3 which I can use for experimenting if you would like me to check if this is a pi zero only issue. As the raspbian os lite comes with very little is it possible I am missing a program or feature which Is essential to neopo? I am willing to try installing Linux on the devices if this would make troubleshooting easier.
Apologies for coming at you with more problems. I really appreciate the help you’re providing.