We just shipped some library refinements to the Web Build IDE system to make faster iteration possible and accommodate a really important use case folks have been asking for: Forking a library. The particular enhancements are:
You can delete a private library or published library if no apps include it.
You can re-import a private library without cutting a new version/release.
You you can fork a published library, tweak it, and use it privately.
To illustrate how these small features work together, I’ll walk through a couple of use cases:
Forking a library in a nutshell
I use a library and discover it doesn’t quite do what I want or has a bug.
I decide to fix it, so I fork it, tweak the code, verify it works as expected, issue a pull request to the original author.
The original author merges in my pull request and re-publishes the library with the fix.
Profit. ( in the community sense )
Forking a library in detail
User A has published uber-library-example, and User B wants to add another example to it to illustrate the incredible things they can do with it ( User B knows how to blink an LED with mad styles)
User B clicks the “fork library” button, forks it on GitHub, clones it locally, adds an example file, commits and pushes it, comes back to the IDE and clicks “contribute library” and punches in the new GitHub URL (i.e. user_b/uber-library-example), then hits import
User B clicks on the new example they added, and clicks “use this example”, then hits the “flash button” to test out their brilliant creation.
User B jumps up and down because their example works as expected on the Core.
User B issues a Pull Request to User A on GitHub.
User A merges in the Pull Request, edits the spark.json to bump the version (i.e. “0.0.7” to “0.0.8”), commits, and pushes it.
User A goes to the IDE, clicks “contribute library”, punches in their repo, and hits “import”, then hits “publish”.
User B deletes their private forked library, and switches back to the published one
…and the crowd goes wild.
Iteratively improving a library
Most of the time when your making refinements or developing a library, you’ll want to develop, check, debug, and repeat until your software behaves as expected. Up until now this process has been cumbersome in the IDE. This is why we’ve introduced the “re-import” process, which simply replaces all of the existing library files you had imported previously with files from the most recent commit. The feature is available for private libraries only, to use:
Click on the private library you want to re-import files for.
Make changes in Git, commit, and push them.
Click the “re-import” button followed by the “import” button.
Click “Include in App” to use the updated library code in your application or if you already have apps that include this private library, you can simply hit verify or compile on that app again to check that it works correctly.
Next time you want to make a change to an existing library in the IDE, please give these features a shot and please share your experiences and questions here!
Thanks for this it will be very useful particularly for the cases mentioned where the primary purpose is adding / improving functionality to the overall library.
Let me add another use case where I think it may not work so well.
Library has some configuration parameters which are not dynamic. I.e. they cannot be set by part of the library API. An example would be defines which control what material inside the library are included in the compile, say fonts in a fonts library.
User wants to include the library and set the static configuration for the intended usage, but the intent is not a long term change to the library.
Now the github forking method could obviously be used to achieve this may be a bit elaborate for this purpose and potentially results in multiple external forks on the library contributor with no intention to issue pull requests. In a classical IDE this would typically be achieved by setting up preprocessor definitions in the IDE itself but as far as I am aware this is not possible here.
I can see that having the one general mechanism to cover all possibilities has some desirable characteristics and will use this mechanism if that is the way to go.
Right on @bobtidey , thanks for bringing this up, developer work flows are always a tricky thing to explain, interpret, and implement. I can definitely see situations where a user might need to unnecessary fork a library to make a change (i.e. “when a library has some configuration parameters which are not dynamic”), and that’s no good. When this happens, though, I’m thinking that is more a reflection of the architecture of the library than the library system itself.
For example, if a library author hard codes some values that should be overridable via DEFINES to the user of the library but doesn’t–the library should be refactored to accommodate this and re-released. If a user were forking a library to do this override, it’d be much better if they did this refactoring then issued a pull request as opposed to forking it, replacing the values, and never issuing a pull request to eliminate the problem in the first place. For library authors to take this happy path though, we’ll need to discuss the particulars, establish conventions and workflows, and tweak the library system itself where needed to maximize library reusability and ease of use.
On that note, if you’ve got a particular code example that illustrates the problem your thinking of, it’d be great if you could share it, and we can discuss it more deeply to figure out the what the right solution is.
One example that inspired the comment was peekay’s excellent extension of the Adafruit GFX to mGFX library. This now has a number of font tables, and a set of defines which control which get compiled into the library during the build. Now you can already control which fonts you are going to use via the library calls, that’s not the issue. If you can’t set the defines then you end up with all font tables included in the build and using up valuable memory when a particular app is never going to use all the fonts.
The same type of thing could apply to libraries with optional sections of code that without compile time control will always get included, even though they are never going to be used in a particular instance.
Alternatively, if that doesn't work. I wonder if the library could be refactored to not set the defines inside the library and pull them out into the example firmware files--to expect the user of the lib to always set the define when using the library, instead of my making that assumption inside the library.
I did try something like the external #define with the mGFX library but it didn’t work. The variables in question weren’t actually defined in the library. I think it might possibly work if the #defines are operating on material within the library .h file but here, like in most case, the #defines are operating on material within the library .cpp file. The library includes the library .h file but I don’t think the library cpp file will see these the redefines in the app file at the point when the library cpp file is compiled. This is normal behaviour and is one of the reasons why IDE environment defines are widely used for this type of control.
I guess one could try to refactor libraries to move all material like this into the .h files but that is a bit artificial and might be tricky in cases where the cpp needs to see these.
The second method of setting the defines externally would I think work if the library included a separate header file that the user always supplied. I think peekay suggested something like that for this mGFX library but wasn’t keen to do anything as library forking was upcoming and might produce a better solution.
Another thought is to make the second method more generic by allowing a general header file which if present is automatically included by the IDE when compiling. This could then contain any external defines needed and would be the equivalent of the defines allowed for in other IDEs.
@bobtidey, @jgoggins, the IDE library requirements forces an “un-natural” requirement for these includes. The specific includes are buried in the _mfGFX library and are not exposed to the user program. So, yes for clarity the only real method for doing this is to extract the user-modifiable #defines into a separate header file. HOWEVER, being part of the library “package”, the SAME problem will exist where that file will not be editable!! This is not an issue with the library as much as it is a problem with the read-only nature of the IDE library mechanism.
I agree if the imported library were instance editable the problem would go away.
What I was suggesting as a (not so nice) alternative was that the user would add to the app a separate non-library file with a fixed name that would be included if present by the IDE in all compiles and be equivalent to IDE variables. This would not require any library changes.
Hey @bobtidey@peekay123 , I think you guys are onto something. For private, forked libraries, I can see little reason why they shouldn’t be editable in the IDE. This would be useful for folks to be able to start hacking on a fix/tweak right way, instead of having to clone locally, commit, push, and re-import. I’ve added this to the backlog, will ping you when we put some dev into this.
I really like the idea of @bobtidey to have that fixed-name header file that may be present in a certain project or not and if it exists, it will be included in all library code files.
There are two reasons that make this my #1 approach:
Even if I have forked a library and included it for private use, I always have to change the library for different purposes (e.g. different OLED displays in different projects). Even that these lib- modifications are per-instance, it would be much cleaner to have that fixed-name header file that is part of the project - and thus, is naturally project specific - and leave the library completely untouched (read-only).
Second reason that this is a well known approach since most of the well-known IDEs have kind of a “project-properties” dialog that allows to specify project-level defines. The fixed-name header file would mimic the exact same behavior .
Jul’14 means July 2014. Thought is 14th of July 2016. But seems to be an open issue since i struggle with the exact same problem today in 2016
If I include a Particle community library in my app (using Web IDE), and if I do not change the names of the included library .h and .cpp files, does my app’s “copy” of the Particle library files ever automatically update if the original Particle library files are updated? If I understand this forum thread, the answer is “no, I would have to manually pull a change” - but I want to be sure.
Yup, that’s how it works.
So even if there were breaking changes to the library, your app would still stay intact since it has the working copy of the library linked.
)
Jul’14 means July 2014. Thought is 14th of July 2016. But seems to be an open issue since i struggle with the exact same problem today in 2016 
