TLDR; I would suggest either avoiding the RC tag, or using it more in line with norms for software development.
Over at 0.7.0 Release Changelog, Please?, it was pointed out that the changelog for firmware 0.7.0 was at https://github.com/particle-iot/firmware/blob/develop/CHANGELOG.md#070-see-additional-changelog-070-rc1--070-rc7. When I read through that, I began to understand my confusion about why 0.7.0 was in release candidate stage for so long. My conclusion is that Particle is not using the normal alpha–>beta–>release candidate naming convention.
The normal convention, as I understand it (and roughly mirrored on wikipedia) is that a new software version flows first from alpha, when new features are added; to beta, when there is a feature freeze and the ensemble is tested for bugs; and then finally to release candidate, when the devs think the release is done and all that is left is widespread testing for edge cases.
In that SRLC world, the RC stage is not when new features would be introduced or APIs deprecated. By RC stage-- because we typically expect little to no changes which would affect stability-- the message sent to the clients (us) is that it’s safe to begin testing for deployment across our entire fleet.
Personally, I would expect changes in the RC phase to be only bugs caught in widespread testing, e.g. tiny things like obscure Android interactions or fixing typos. Anything larger mean our testing results are no longer valid and we need to restart.
Not complaining about the release cycle or dev efforts, just observing that reusing the common nomenclature in this different way leads to confusion. For instance, to the casual observer it looks like it took Particle almost a year to get 0.7.0 out of RC stage. In reality, if we consider that RC started with 0.7.0-rc.6, then it was a reasonable 3 months to work the final edge case bugs out.
Do the devs agree? If not, what is the driving philosophy behind the naming convention used at Particle?
2 Likes
You are correct, about the normal use of RC and Particle would usually not have that many RCs before a release.
With 0.7.0 conincidence would have it that just before an official realase was planned something else came in that needed to be incorporated (e.g. WICED stack changed, impacting the newly added WPA Enterprise, the Krack exploit was revealed and had to be dealt with, these also caused “regressions”, features like WPA Enterprise with limited audience but very heterogenous setups for testing, so bug reports come back slowly, …).
That was just a quick recap of what I hear of Particle, but I hope an official reply will follow too.
I believe the convention here is to bump the minor version number and go back to beta. Harry Hindsight being 20/20, it’s easy to say now that Particle could have done this, but it probably was a much harder call at the time. I’m sure that to many the final release looked just around the corner.
However, where does this leave us for 0.8.0-rc.3? Is was released a little over a week ago, and looking at the changelog it still seems not to have undergone a feature freeze. Some of those changes go to the very heart of the networking stack, which is one of the most critical parts of the code.
It might just be that Particle is using a more web-tech-based model of feature releases, where they just roll out non-stop. I can get behind that, so long as it’s cleary what to expect. I love new features as much as the next fellow, and I’d feel much more comfortable justifying my firmware decisions to my company principals if I understood the core dev’s philosophy on naming conventions.
That’s where an official statement would be good to follow 
@Joe, can you chime in on that?
3 Likes
Great questions @kubark42 — thanks so much for the openness and curiosity.
Summary:
- The only semver pre-release identifier we add to version numbers is
rc (no alpha, beta, etc.).
- We do think of release candidates as "going silver", potentially final product, though we know from experience that
rc.1 is never final.
- Thus, we have not, in the past, always formally frozen features, but any additional functionality consistently gets a serious internal debate. When we add features to release candidates, it's often due to a specific customer request.
- We'll keep your perspective in mind going forward and more clearly communicate when we've hit feature freeze (which might not be
rc.1).
I think at the end of your most recent post, you hit the crux of the philosophy—or at least our starting point of view.
We are, in fact, trying to release the Device OS as frequently as possible (following the web-tech example of the rest of the Particle stack). Ideally, we'd like to be shipping new Device OS versions every sprint (3 weeks). At least for now, that's pie in the sky.
As you're no doubt aware, in firmware land, the testing process in particular takes longer. We automate whatever testing we can, but some manual testing with several varieties of hardware will always be required, and often the bugs that arise during testing are a lot more esoteric and difficult to uncover, reproduce, and fix than in web software.
Also, we've iterated on this process and continue to learn how our theoretical ideas actually function in the very human meatspace of the 
community. Some time last year we started measuring compiles by firmware version. We learned that despite our attempts to solicit community testing, essentially everyone just compiles against whatever is default. You can see it in this graph of activity on March 26 when we made 0.7.0 default—blue is 0.6.3 compiles, red is 0.7.0 compiles.
We'd like to see more red on the left side. We are having discussions internally about how to increase community adoption and testing of release candidates and how to get more feedback.
A related question came up internally about the 0.8.0-rc.4 feature set and timing today. There are always these opposing forces with every release: "We need this feature now!" vs "Slow down, test more, focus on stability." Our goal is to have the genuine, honest conversations every time, hearing both sides for any given feature/release, and to strike the right balance: on one hand unblocking customers rapidly to give them the competitive edge in IoT, and on the other hand continuing to build on our reputation for stability.
9 Likes
Thanks, @zachary. That is a very clear and cogent explanation. In light of it, might I suggest that Particle chose a pre-release identifier other than RC flag? RC carries a lot of baggage with it, and doesn’t really seem to be an appropriate fit for Particle. Perhaps Particle could invent its own pre-release identifier?
I always want to be careful about redefining standards and conventions, because not only is it confusing to those looking at my projects, it also weakens the convention outside my project (I think of it as a similar mechanism to herd immunity). Maybe what are termed “releases” right now could be MTS (medium-term-support) instead, and normal releases could use a convention like Windows has (Microsoft is currently at 10.0.16299.402 for Windows 10.).
I’m excited to hear about a plan to roll out tri-weekly releases. However, to the best of my knowledge, Particle is the only company which is selling a product with this firmware approach. (We did this with our autopilot, but we were giving it away for free.) I love that you’re innovating, “Run fast and break things” is a great way to make sure that small bugs are caught immediately, instead of 6 months down the road when the code isn’t as fresh in the developer’s mind.
On the other hand, it’s bucking a tried-and-true firmware approach. Has Particle had a chance to look back and decide if this is working, both for the company and its clients? If not, is there a date at which Particle will decide if this experiment is working?
Lastly, world-class documentation becomes even more important when new features are rolled out weekly. Somewhere, resolving this kind of situation, https://github.com/particle-iot/docs/pull/760, takes on even more importance when everything moves so quickly. Maybe a rule such as “known bugs are either documented in the official documentation or fixed in the next sprint” is appropriate. This seems like a fair compromise between not overloading docu managers and not leaving clients to rediscover bugs.
1 Like
This is only one data point, but the reason why I did not use 0.7.0-rc7 for development was because I couldn't understand if that was expected to be a stable version or not. I noticed that 0.7.0 had been in RC for the better part of a year, and I couldn't tell if it were already obsoleted by 0.8.0. The concern was that the entire 0.7.0 branch might have had fatal bugs causing it to stall out in RC, with focus being turned to 0.8.0.
Thank you; great thoughts. We constantly look back and ask ourselves what’s working and what could be better. I’ll leave further discussion of Particle’s position and any experiments we might want to try to the firmware team including @mdma.
1 Like
