Building latest code for the Core

Continuing the discussion from Particle Matter: Updates from Sprint 39:

The develop / latest branches also have firmware for the core. You build it using

make all

from the main folder. More details in the build docs.

1 Like

Iā€™ve installed GitHub, ARM Embedded 4.9, make (GnuWin32), etc. Downloading the ā€œlatestā€ branch as a ZIP gets me the latest files (a lot of HAL stuff, etc.)ā€¦but I get fail compile errors.

OK, follow the instructions on the ā€œlatestā€ branch: https://github.com/spark/firmware/tree/latest

The ā€œlatestā€ branch readme says to either download as ZIP, or ā€œgit clone https://github.com/spark/firmware.gitā€. That gets meā€¦the OLD firmware version! On top of that, it does not include the rest of the projectā€“basically, Iā€™m back to the instructions on ā€œmasterā€, which requires a download of ā€œfirmware.gitā€ as well as ā€œcore-common-libā€ and ā€œcore-communication-libā€. And that means that Iā€™m right back at what I get when I push the ā€œFlash with Cloudā€ button, minus a lot of time.

OK, so I download the ā€œlatestā€ branch as a ZIP, extract it to a folder. Following the instructions on the ā€œlatestā€ branch page, I navigate to the ā€œmainā€ folder, and type ā€œmake.ā€ Fail, fail, failā€¦hereā€™s the command prompt output:

D:\ParticleSrc\firmware-latest>cd main

D:\ParticleSrc\firmware-latest\main>make
fatal: Not a git repository (or any of the parent directories): .git
'tr' is not recognized as an internal or external command,
operable program or batch file.
'tr' is not recognized as an internal or external command,
operable program or batch file.
'tr' is not recognized as an internal or external command,
operable program or batch file.
'head' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the file specified.
make -C ../user
fatal: Not a git repository (or any of the parent directories): .git
'head' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the file specified.
make[1]: Entering directory `D:/ParticleSrc/firmware-latest/user'
'Building file: src/application.cpp'
'Invoking: ARM GCC CPP Compiler'
mkdir -p ../build/target/user/platform-0-lto/src/src/
The syntax of the command is incorrect.
make[1]: *** [../build/target/user/platform-0-lto/src/src/application.o] Error 1

make[1]: Leaving directory `D:/ParticleSrc/firmware-latest/user'
make: *** [user] Error 2

D:\ParticleSrc\firmware-latest\main>

To the best of my knowledge, I have everything installed, PATHed correctly (I can call any of the GCC compilers from the command prompt, MAKE is found, etc.) What is going on?

EDIT: Try ā€œmakeā€ from the root folder. Looks a little better, but still fails:

D:\ParticleSrc\firmware-latest\main>cd ../

D:\ParticleSrc\firmware-latest>make
Building firmware for Spark core, platform ID: 0, product ID: 0
make -C ./communication
fatal: Not a git repository (or any of the parent directories): .git
'head' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the file specified.
make[1]: Entering directory `D:/ParticleSrc/firmware-latest/communication'
'Building file: lib/tropicssl/library/aes.c'
'Invoking: ARM GCC C Compiler'
mkdir -p ../build/target/communication/platform-0-lto-prod-0/lib/tropicssl/libra
ry/
The syntax of the command is incorrect.
make[1]: *** [../build/target/communication/platform-0-lto-prod-0/lib/tropicssl/
library/aes.o] Error 1
make[1]: Leaving directory `D:/ParticleSrc/firmware-latest/communication'
make: *** [communication] Error 2

D:\ParticleSrc\firmware-latest>

EDIT 2: Iā€™m running Windows 7 Ultimate x64.
P.S. Command-line compilers and me donā€™t seem to mix very well.


(ScruffR: Reformatted text block)

 ```text
 like this

Found the problem. It had to do with the GitHub installer NOT properly setting its path. (This is a local build: why do I need GitHub?) 'Twas setting it to ā€œD:\ParticleBuild\Git\cmdā€, a folder which conveniently ONLY contained ā€œgit.exeā€. I modified it to ā€œD:\ParticleBuild\Git\binā€ā€¦and now after 30 seconds and pages (and pages) of compiler output, I have some legitimate errors. For what itā€™s worth, I havenā€™t touched the extracted files at allā€“if this is supposed to fail, then itā€™s doing what itā€™s supposed to:

<extreme excerpt from the very bottom of the output>

Invoking: ARM GNU Create Flash Image
arm-none-eabi-objcopy -O binary ../build/target/bootloader/platform-0-lto/bootlo
ader.elf ../build/target/bootloader/platform-0-lto/bootloader.bin.pre_crc
if [ -s ../build/target/bootloader/platform-0-lto/bootloader.bin.pre_crc ]; then
 \
        head -c $((`stat --print %s ../build/target/bootloader/platform-0-lto/bo
otloader.bin.pre_crc` - 38)) ../build/target/bootloader/platform-0-lto/bootloade
r.bin.pre_crc > ../build/target/bootloader/platform-0-lto/bootloader.bin.no_crc
&& \
        tail -c 38 ../build/target/bootloader/platform-0-lto/bootloader.bin.pre_
crc > ../build/target/bootloader/platform-0-lto/bootloader.bin.crc_block && \
        test "0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f2028
0078563412" = `xxd -p -c 500 ../build/target/bootloader/platform-0-lto/bootloade
r.bin.crc_block` && \
        ../build/bin/win32/sha256sum ../build/target/bootloader/platform-0-lto/b
ootloader.bin.no_crc | cut -c 1-65 | xxd -r -p | dd bs=1 of=../build/target/boot
loader/platform-0-lto/bootloader.bin.pre_crc seek=$((`stat --print %s ../build/t
arget/bootloader/platform-0-lto/bootloader.bin.pre_crc` - 38)) conv=notrunc  &&
\
        head -c $((`stat --print %s ../build/target/bootloader/platform-0-lto/bo
otloader.bin.pre_crc` - 4)) ../build/target/bootloader/platform-0-lto/bootloader
.bin.pre_crc > ../build/target/bootloader/platform-0-lto/bootloader.bin.no_crc &
& \
         crc32 ../build/target/bootloader/platform-0-lto/bootloader.bin.no_crc |
 cut -c 1-10 | xxd -r -p | dd bs=1 of=../build/target/bootloader/platform-0-lto/
bootloader.bin.pre_crc seek=$((`stat --print %s ../build/target/bootloader/platf
orm-0-lto/bootloader.bin.pre_crc` - 4)) conv=notrunc ;\
        fi
/bin/sh: line 1: stat: command not found
head: -38: invalid number of bytes
make[1]: *** [../build/target/bootloader/platform-0-lto/bootloader.bin] Error 1
make[1]: Leaving directory `D:/ParticleSrc/firmware-latest/bootloader'
make: *** [bootloader] Error 2

D:\ParticleSrc\firmware-latest>

More of the same fun, I assume? Whereas I could find ā€œhead.exeā€ in the Git folder, I canā€™t seem to find anything called ā€œstat.exeā€. Now what?

1 Like

Tried the above example, doing a compile with MSYS instead of GnuWin32. All that changes is that HEAD.EXE and TR.EXE outright crash, and it can't find a valid ARM GCC version. No go.

EDIT: I can successfully compile the "Master" branch with GnuWin32, giving me a 78KB BIN file. Trying again with "firmware-latest/main > MAKE", I realize that I'm actually getting an output file, buried in "firmware-latest\build\target\main\platform-0-lto\main.bin.pre_crc", from OBJCOPY. Definitely looks different inside, but still looks like a code file (76K inside). Strings like, "Restarting system to apply firmware update", "Firmware update using this terminal is not supported!", "Add #include "Ymodem/Ymodem.h" to your sketch and try again.", "The file size is higher than the allowed space memory!", etc. If the compiler output is to be believed, it appears that I'm just missing a CRC check?

I have "head.exe", "tail.exe", but not "test.exe", "crc32.exe", or "stat", whatever that is.

Is anyone able to successfully build the "Latest" branch locally for the Core, on Windows 7 x64? I'm using the latest downloads, which could be the problem--I just don't know what to do.

@ScruffR, if you have answers, please let @mdma enjoy his vacation :wink:

OK. Errors and all, never mind that the compile never finishes. Figured out how to steal the output BIN from deep in the folder treeā€¦like that itā€™s 98K with same code that produces 103K from Particle Dev. Really like the TCPServer.stop call (it wasnā€™t TCPServer.close after all), WiFi.resolve() in the Coreā€¦you can tell Iā€™m determined :wink:

All dandy (oh yes, redoing my files to CPP format so theyā€™d show up and compile)ā€¦exceptā€¦I wonder if thereā€™s a more relevant thread for debug build testing?

At any rate, WiFi.ping() always returns 0 with the ā€œfirmware-latestā€ branch. WiFi.resolve returns a proper IP address (which I previously had to get with ā€œgethostbynameā€). I pass that IP address to WiFi.ping(), which returns 0 after a couple of seconds. The returned IP addresses are the same as I get from the Particle Dev buildā€“in which WiFi.ping() gives a positive response.

1 Like

Maybe Iā€™m just writing this thread for myself :blush:. ANYWAYā€¦still havenā€™t gotten the LATEST branch to compile without errors. I can follow the tutorials and compile the ā€œFIRMWAREā€ branches with no problem, so Iā€™m suspecting something wrong with the makefile. (ā€œThat line donā€™t work? Remove it!ā€) But Iā€™m not complainingā€“Iā€™ve found the output file, and the 1.7MB linker output.

Now to the tweakingā€¦by removing all the libraries I wasnā€™t using (in ā€œfirmware-latest\wiring\srcā€), Iā€™ve saved a total of 1,764 bytes on the output file. Now, I can do a lot with that! Libraries Iā€™ve removed by renaming to ā€œ.CPP.Xā€ or ā€œ.H.Xā€ (probably not the correct way, but heyā€“it works!) include:

  • EEPROM (Iā€™m using FlashEE anyway :wink: @mdma)
  • I2C
  • Servo
  • SPI (FlashEE uses a Core-specific #include)
  • Tone
  • UDP
  • USART Serial
  • USB Keyboard
  • USB Mouse
  • WiFi Tester

Oh yes, then removing calls and #includes from files including ā€œspark_wiring.hā€ that referenced particularly SPI, I2C and USART Serial. (This was probably where my gains came from.) Tedious? Yes.

For the record, Iā€™m using the amazingly comprehensive PSPad as a developer GUI. I took a look at NetBeansā€™ 700MB install size, and said, ā€œNo way.ā€ PSPad installs without any voliminous Java dependencies, and is only 15MB in size. Do I miss the polished professionalism of Particle IDE? Yes. Do I miss the ease of use? Sure. Worth it? Absolutely!

Trying to get rid of the compile errors the brute force way:

  • Commented out a reference to ā€œchecks.mkā€ from ā€œfirmware-latest\build\common_tools.mkā€ā€“got rid of all those fatal: Not a git repository (or any of the parent directories): .git errors
  • Completely stumped on how to get rid of the ā€œstat not foundā€ error. If I give it Stat.exe from MSYS 1.0, Stat crashes, and I get a stack dump. Not much of a solutionā€¦

Since switching to the LATEST branch on a local build (despite the continuing compiler issues), the output BIN (well, actually ā€œ.bin.pre_crcā€) is 4,616 bytes smaller than the Particle Dev IDEā€™s outputā€¦and I havenā€™t removed anything from my code.
Everything works, with the sole exception of WiFi.ping(); It takes the two-second block, and returns 0. TCPClient can connect just fine to the same servername and communicate both ways, and WiFi.resolve() returns a valid IP address. So far, it appears that just WiFi.ping() is broken.

EDIT: The code for WiFi.ping() is practically the same between ā€œLATESTā€ and ā€œCOMPILE-SERVER2ā€. @mdma, I see your name on the source code file :wink:. Two lines are different, namely uint32_t pingIPAddr = ip->ipv4; and system_tick_t lastTime = HAL_Timer_Get_Milli_Seconds(); stuffā€¦but the timer stuff is working, as it blocks ~2 seconds each time. I donā€™t see why it isnā€™t working. But it isnā€™t.

1 Like

Whew...finally found a solution on the forum for STAT.exe crashing! (I do feel all alone out here, though :persevere:)

Bingo! No crash, no error, and I get an output that I don't quite understand:

   text       data        bss        dec        hex    filename
  97592        832      10076     108500      1a7d4    ../build/target/main/platform-0-lto/Project.elf

The BIN file is 98,428 bytes. But if the "DEC" value is to believed, I'm just about out of memory (according to @mdma, 110,592 bytes max.) But DFU-Util downloads the 98,428 bytes of the BIN file, not the 108,500 of the "stat" output.

WiFi.ping() is just simply, plainly broken. I don't understand how, as the changes don't seem major. Serial.println(WiFi.ping(WiFi.gatewayIP())); returns zero. Every time. I have confirmed that the IP address is correct in "wlan_hal.c", where the inet_ping function is found...and even tried reversing the sequence of the IP address bytes (3210 -> 0123). The function uses netapp_ping_send (found in "netapp.c"), which has not changed at all.

@WebDust21, the ā€œlatestā€ branch is really for the Photon and it is early days for the new Core firmware. I have found that code which fit in flash in the 0.3.4 firmware environment no longer fits with the HAL code. The focus right now is really on the Photon and the Core update still needs lots of lovinā€™. :smile: