Hey,
My team and I experience about a 80% failure rate on cloud compiles. We always get the error:
attempting to compile firmware
Compile failed. Exiting.
Server error
make: *** [bin/firmware.bin] Error 1
We just keep trying and eventually it succeeds. What might the cause of these intermittent failures be and why are the failing so often?
@incorvia, could you please also state what way you are cloud compiling (Web IDE, Particle Dev, CLI)?
If you are on one of the latter two, your project folder might contain conflicting files that break the build (e.g. source files containing setup() and/or loop())
Also what firmware version are you targeting?
Sometimes one target fails but another works - we’d need to know as exact as possible.
@ScruffR Thanks for helping me investigate.
We are using the particle-cli largely.
We were targeting PARTICLE_VERSION=0.6.1 but we seem to have the same experience when just targeting the latest.
Also I want to clarify. The same code succeeds after failing 4-5 times in a row. This is our experience. So I’m curious, if it’s an issue with project folder containing conflicting files, why would it fail, fail, fail, fail, fail, succeed? Do you think it has to do with the way the files are being globbed by the CLI and the order they are being included?
I can add that if the compile is supposed to legitimately fail, it seems to report the proper errors. Its only when I start getting this compile failed. exiting. error that I know it likely will succeed if I keep retrying it.
@bryce Please let me know if you need any more information to help diagnose. It’s very problematic testing our code since we have to wait at times 5 - 10 minutes to get a successful compile.
@incorvia I’m looking into this now. Could you see if there a the same issues with 0.6.1-rc.1?
Seems to be some compile errors on 0.6.1-rc.1… Trying to see what the deal is. Is it common to get different errors across targets?
It might point to something about the sources. Recently we changed the compile service infrastructure and I wanted to see if the issue lays there (0.6.1-rc.1 is using the old infrastructure). It seems that your compiles are on the edge of timing out so with low load they pass. Is it possible for you to share the code with us?
@suda, what is the timeout set at? I’d suspect that you must be targeting allowing a certain number of deviations of builds through. Would that mean we would be greater than X percentile of build times? As for sharing the code I’ll need to discus with team.
I think the default should be 2 minutes and it’s for each job. By load I meant more compile jobs coming from other users at the same time which may increase build time of your jobs. I understand if you can’t share the code. Could you PM me the username (email) being used for the compilation?
Same thing here:
I’m using the CLI to compile and I’m compiling with this command:
particle compile photon --target 0.6.0
attempting to compile firmware
Compile failed. Exiting.
Server error
It’s very annoying…
Same thing with me. Usually it happens 2-3 times in a row before actually succeeding. This means that every time I go to compile, I’m typically spending about 5-8 minutes… this is killing my productivity!!!
I’m guessing the particle compile farm isn’t designed for larger apps, as it seems that both my app and @incorvia both have lots of files getting uploaded. I can confirm that I am having no errors ever when compiling smaller projects.
I wouldn’t be surprised if the build traffic grows exponentially as this problem begins to become an issue for more and more developers as their projects get bigger. Think hundreds of developers getting compile timeouts in Particle CLI and just sitting there re-initiating the request each time it times out… starts to snowball pretty quickly with that kind of positive feedback loop!
It would appear that the compile is returning the server error message exactly 1 minute after Particle CLI says that its uploading the files. So, this doesn’t jive with the 2 minute timeout that @suda is talking about above.
Unfortunately I can’t use the Particle Dev method of compiling now that Particle Dev has gone the route of “library manager” features, which effectively break compiling of old-style file structures (see my post on this here)
@suda: Can you look into my info like you did for @incorvia please?
Here is what I get with Particle-CLI:
C:\Users\Boompy\Documents\Bitbucket\JazaHub Controller\mainMash>particle compile electron --target 0.6.1
Compiling code for electron
Targeting version: 0.6.1
Including:
main.cpp
AFE\DataManager.h
AFE\jazaAFE.h
AFE\MAX11610_registers.h
AFE\MAX11610.h
AFE\systemData.h
CAN\bitMasks.h
CAN\CM_CAN.h
CAN\jazaCAN_config.h
CAN\jazaCAN.h
Charging\jazaHubChannel.h
Charging\jazaPackManager_config.h
Charging\jazaPackManager.h
Charging\transactionDataTypes.h
Control\JazaControl.h
Network\CellularCredentials.h
Network\IMSI.h
Network\NetworkManager.h
PCB\jazaPinout.h
Publishing\JazaCloud.h
Publishing\JazaPublish.h
SD\SdFat\ArduinoFiles.h
SD\SdFat\ArduinoStream.h
SD\SdFat\bufstream.h
SD\SdFat\FatApiConstants.h
SD\SdFat\FatFile.h
SD\SdFat\FatFileSystem.h
SD\SdFat\FatLib.h
SD\SdFat\FatLibConfig.h
SD\SdFat\FatStructs.h
SD\SdFat\FatVolume.h
SD\SdFat\FmtNumber.h
SD\SdFat\FreeStack.h
SD\SdFat\fstream.h
SD\SdFat\ios.h
SD\SdFat\iostream.h
SD\SdFat\istream.h
SD\SdFat\ostream.h
SD\SdFat\SdFat.h
SD\SdFat\SdFatConfig.h
SD\SdFat\SdInfo.h
SD\SdFat\SdSpi.h
SD\SdFat\SdSpiCard.h
SD\SdFat\SoftSPIParticle.h
SD\SdFat\StdioStream.h
SD\SdFat\SysCall.h
SD\SdFat\SystemInclude.h
Reboot\RebootManager.h
SD\EscapeChars.h
SD\JazaSD.h
UI\jazaBitmaps.h
UI\SparkFunSX1509.h
UI\ST7565.h
UI\sx1509_registers.h
UI\userInterface.h
AFE\DataManager.cpp
AFE\jazaAFE.cpp
AFE\MAX11610.cpp
AFE\systemData.cpp
CAN\CM_CAN.cpp
CAN\jazaCAN.cpp
Charging\jazaHubChannel.cpp
Charging\jazaPackManager.cpp
Control\JazaControl.cpp
Network\CellularCredentials.cpp
Network\IMSI.cpp
Network\NetworkManager.cpp
Publishing\JazaCloud.cpp
Publishing\JazaPublish.cpp
SD\SdFat\FatFile.cpp
SD\SdFat\FatFileLFN.cpp
SD\SdFat\FatFilePrint.cpp
SD\SdFat\FatFileSFN.cpp
SD\SdFat\FatVolume.cpp
SD\SdFat\FmtNumber.cpp
SD\SdFat\fstream.cpp
SD\SdFat\istream.cpp
SD\SdFat\ostream.cpp
SD\SdFat\SdFat.cpp
SD\SdFat\SdSpiCard.cpp
SD\SdFat\SdSpiParticle.cpp
SD\SdFat\StdioStream.cpp
Reboot\RebootManager.cpp
SD\EscapeChars.cpp
SD\JazaSD.cpp
UI\SparkFunSX1509.cpp
UI\ST7565.cpp
UI\userInterface.cpp
UI\glcdfont.c
attempting to compile firmware
Compile failed. Exiting.
Server error
_
_
_
Also: not sure if this helps but here are my particle.include and particle.ignore files:
Particle include:
#particle.include
# from current dir
*.h
*.ino
*.cpp
*.c
# and from any subdirectories
**/*.h
**/*.ino
**/*.cpp
**/*.c
Particle ignore:
#particle.ignore
# from current dir
# and from any subdirectoriesSure! I made a script that uses the same components our cloud compiler does: compile.sh. It runs our buildpacks (Docker images containing toolchain) on your local Docker (if you don’t have it installed you can find it here: Docker for Windows and Docker for Mac). Please let me know if you have any issues with it.
Thanks for the help @suda !
Couple of questions:
I’m new to Docker, so am bound to make many a mistake trying to get this up and running. I was able to run the script you linked to by downloading compile.sh to my desktop, opening up a docker terminal, and typing ./compile.sh
Here is the result:
Please set ACCESS_TOKEN env variable
@jaza_tom, I use ZSH so I just added my particle access token to my env via my .zshrc file but if you use bash you could do the same:
http://www.mcsr.olemiss.edu/unixhelp/environment/env3d.html
As for what the token you can use the CLI: “particle token new” to generate it.
Also I just run the command ./compile.sh no need to run it with the docker terminal. Not sure if docker terminal is something else as I haven’t used that before.
Another gotcha is you need to use all relative includes for headers as files don’t appear to be flattened. But I assure you I have the local compiles with docker working and it’s worth the price of admission.
Hey, so I’m going to flip out pretty soon. I’ve just spent 9 hours trying to get a local compiling setup to work and trying to use this docker stuff.
Can’t get it to work.
See here for my local compiling woes
I’m getting quite fed up.
Any plans to fix the Particle server timeout so that I can just use cloud compiling? Can anyone comment on this @suda @mdma @ScruffR ?
I’m sorry to hear that it didn’t work. Can you give more background what didn’t work with Docker for Windows? We just updated our particle-dev-local-compiler package for Atom which might be easier to use than the script. As for the cloud compiler, we will continue improving its performance but at certain project size it might not be as effective to use it (for every compilation a whole project needs to be sent to our servers which add even more time to the process). I’m happy to working with you on improving your experience here!
I have offline compiling now workign with po-util so I should be good to go. As far as the Docker problem, I think it was just my ineptitude since I’ve never used it before. Thanks!