Building latest code for the Core

[mdma]

Continuing the discussion from Particle Matter: Updates from Sprint 39:

@WebDust21

Any news on 0.4.0 for the Core?

EDIT: I go looking to local-build a development branch for the Core, and can't find any. The Photon (develop) branch has been updated within the past few days; the Core (master) branch dates from months to years back.

The develop / latest branches also have firmware for the core. You build it using

make all

from the main folder. More details in the build docs.

[WebDust21]

I’ve installed GitHub, ARM Embedded 4.9, make (GnuWin32), etc. Downloading the “latest” branch as a ZIP gets me the latest files (a lot of HAL stuff, etc.)…but I get fail compile errors.

OK, follow the instructions on the “latest” branch: https://github.com/spark/firmware/tree/latest

The “latest” branch readme says to either download as ZIP, or “git clone https://github.com/spark/firmware.git”. That gets me…the OLD firmware version! On top of that, it does not include the rest of the project–basically, I’m back to the instructions on “master”, which requires a download of “firmware.git” as well as “core-common-lib” and “core-communication-lib”. And that means that I’m right back at what I get when I push the “Flash with Cloud” button, minus a lot of time.

OK, so I download the “latest” branch as a ZIP, extract it to a folder. Following the instructions on the “latest” branch page, I navigate to the “main” folder, and type “make.” Fail, fail, fail…here’s the command prompt output:

D:\ParticleSrc\firmware-latest>cd main

D:\ParticleSrc\firmware-latest\main>make
fatal: Not a git repository (or any of the parent directories): .git
'tr' is not recognized as an internal or external command,
operable program or batch file.
'tr' is not recognized as an internal or external command,
operable program or batch file.
'tr' is not recognized as an internal or external command,
operable program or batch file.
'head' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the file specified.
make -C ../user
fatal: Not a git repository (or any of the parent directories): .git
'head' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the file specified.
make[1]: Entering directory `D:/ParticleSrc/firmware-latest/user'
'Building file: src/application.cpp'
'Invoking: ARM GCC CPP Compiler'
mkdir -p ../build/target/user/platform-0-lto/src/src/
The syntax of the command is incorrect.
make[1]: *** [../build/target/user/platform-0-lto/src/src/application.o] Error 1

make[1]: Leaving directory `D:/ParticleSrc/firmware-latest/user'
make: *** [user] Error 2

D:\ParticleSrc\firmware-latest\main>

To the best of my knowledge, I have everything installed, PATHed correctly (I can call any of the GCC compilers from the command prompt, MAKE is found, etc.) What is going on?

EDIT: Try “make” from the root folder. Looks a little better, but still fails:

D:\ParticleSrc\firmware-latest\main>cd ../

D:\ParticleSrc\firmware-latest>make
Building firmware for Spark core, platform ID: 0, product ID: 0
make -C ./communication
fatal: Not a git repository (or any of the parent directories): .git
'head' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the file specified.
make[1]: Entering directory `D:/ParticleSrc/firmware-latest/communication'
'Building file: lib/tropicssl/library/aes.c'
'Invoking: ARM GCC C Compiler'
mkdir -p ../build/target/communication/platform-0-lto-prod-0/lib/tropicssl/libra
ry/
The syntax of the command is incorrect.
make[1]: *** [../build/target/communication/platform-0-lto-prod-0/lib/tropicssl/
library/aes.o] Error 1
make[1]: Leaving directory `D:/ParticleSrc/firmware-latest/communication'
make: *** [communication] Error 2

D:\ParticleSrc\firmware-latest>

EDIT 2: I’m running Windows 7 Ultimate x64.
P.S. Command-line compilers and me don’t seem to mix very well.

---

(ScruffR: Reformatted text block)

 ```text
 like this

[WebDust21]

Found the problem. It had to do with the GitHub installer NOT properly setting its path. (This is a local build: why do I need GitHub?) 'Twas setting it to “D:\ParticleBuild\Git\cmd”, a folder which conveniently ONLY contained “git.exe”. I modified it to “D:\ParticleBuild\Git\bin”…and now after 30 seconds and pages (and pages) of compiler output, I have some legitimate errors. For what it’s worth, I haven’t touched the extracted files at all–if this is supposed to fail, then it’s doing what it’s supposed to:

<extreme excerpt from the very bottom of the output>

Invoking: ARM GNU Create Flash Image
arm-none-eabi-objcopy -O binary ../build/target/bootloader/platform-0-lto/bootlo
ader.elf ../build/target/bootloader/platform-0-lto/bootloader.bin.pre_crc
if [ -s ../build/target/bootloader/platform-0-lto/bootloader.bin.pre_crc ]; then
 \
        head -c $((`stat --print %s ../build/target/bootloader/platform-0-lto/bo
otloader.bin.pre_crc` - 38)) ../build/target/bootloader/platform-0-lto/bootloade
r.bin.pre_crc > ../build/target/bootloader/platform-0-lto/bootloader.bin.no_crc
&& \
        tail -c 38 ../build/target/bootloader/platform-0-lto/bootloader.bin.pre_
crc > ../build/target/bootloader/platform-0-lto/bootloader.bin.crc_block && \
        test "0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f2028
0078563412" = `xxd -p -c 500 ../build/target/bootloader/platform-0-lto/bootloade
r.bin.crc_block` && \
        ../build/bin/win32/sha256sum ../build/target/bootloader/platform-0-lto/b
ootloader.bin.no_crc | cut -c 1-65 | xxd -r -p | dd bs=1 of=../build/target/boot
loader/platform-0-lto/bootloader.bin.pre_crc seek=$((`stat --print %s ../build/t
arget/bootloader/platform-0-lto/bootloader.bin.pre_crc` - 38)) conv=notrunc  &&
\
        head -c $((`stat --print %s ../build/target/bootloader/platform-0-lto/bo
otloader.bin.pre_crc` - 4)) ../build/target/bootloader/platform-0-lto/bootloader
.bin.pre_crc > ../build/target/bootloader/platform-0-lto/bootloader.bin.no_crc &
& \
         crc32 ../build/target/bootloader/platform-0-lto/bootloader.bin.no_crc |
 cut -c 1-10 | xxd -r -p | dd bs=1 of=../build/target/bootloader/platform-0-lto/
bootloader.bin.pre_crc seek=$((`stat --print %s ../build/target/bootloader/platf
orm-0-lto/bootloader.bin.pre_crc` - 4)) conv=notrunc ;\
        fi
/bin/sh: line 1: stat: command not found
head: -38: invalid number of bytes
make[1]: *** [../build/target/bootloader/platform-0-lto/bootloader.bin] Error 1
make[1]: Leaving directory `D:/ParticleSrc/firmware-latest/bootloader'
make: *** [bootloader] Error 2

D:\ParticleSrc\firmware-latest>

More of the same fun, I assume? Whereas I could find “head.exe” in the Git folder, I can’t seem to find anything called “stat.exe”. Now what?

[WebDust21]

Local development and gdb debugging with NetBeans, a step by step guide

I will be repeating a lot of stuff from this video, but I find a text based guide easier to follow myself. Please refer to the video for details. It was my main source for the bunch of links below. The video is also made for the master branch. The feature/hal branch is easier to use (and the future), so my instructions are based on that. Download and install the GNU Tools for ARM Embedded Processors Download here. I chose the windows installer. During install, choose English. It will be easie…

Tried the above example, doing a compile with MSYS instead of GnuWin32. All that changes is that HEAD.EXE and TR.EXE outright crash, and it can't find a valid ARM GCC version. No go.

EDIT: I can successfully compile the "Master" branch with GnuWin32, giving me a 78KB BIN file. Trying again with "firmware-latest/main > MAKE", I realize that I'm actually getting an output file, buried in "firmware-latest\build\target\main\platform-0-lto\main.bin.pre_crc", from OBJCOPY. Definitely looks different inside, but still looks like a code file (76K inside). Strings like, "Restarting system to apply firmware update", "Firmware update using this terminal is not supported!", "Add #include "Ymodem/Ymodem.h" to your sketch and try again.", "The file size is higher than the allowed space memory!", etc. If the compiler output is to be believed, it appears that I'm just missing a CRC check?

I have "head.exe", "tail.exe", but not "test.exe", "crc32.exe", or "stat", whatever that is.

Is anyone able to successfully build the "Latest" branch locally for the Core, on Windows 7 x64? I'm using the latest downloads, which could be the problem--I just don't know what to do.

@ScruffR, if you have answers, please let @mdma enjoy his vacation

[WebDust21]

OK. Errors and all, never mind that the compile never finishes. Figured out how to steal the output BIN from deep in the folder tree…like that it’s 98K with same code that produces 103K from Particle Dev. Really like the TCPServer.stop call (it wasn’t TCPServer.close after all), WiFi.resolve() in the Core…you can tell I’m determined

All dandy (oh yes, redoing my files to CPP format so they’d show up and compile)…except…I wonder if there’s a more relevant thread for debug build testing?

At any rate, WiFi.ping() always returns 0 with the “firmware-latest” branch. WiFi.resolve returns a proper IP address (which I previously had to get with “gethostbyname”). I pass that IP address to WiFi.ping(), which returns 0 after a couple of seconds. The returned IP addresses are the same as I get from the Particle Dev build–in which WiFi.ping() gives a positive response.

[WebDust21]

Maybe I’m just writing this thread for myself . ANYWAY…still haven’t gotten the LATEST branch to compile without errors. I can follow the tutorials and compile the “FIRMWARE” branches with no problem, so I’m suspecting something wrong with the makefile. (“That line don’t work? Remove it!”) But I’m not complaining–I’ve found the output file, and the 1.7MB linker output.

Now to the tweaking…by removing all the libraries I wasn’t using (in “firmware-latest\wiring\src”), I’ve saved a total of 1,764 bytes on the output file. Now, I can do a lot with that! Libraries I’ve removed by renaming to “.CPP.X” or “.H.X” (probably not the correct way, but hey–it works!) include:

• EEPROM (I’m using FlashEE anyway @mdma)
• I2C
• Servo
• SPI (FlashEE uses a Core-specific #include)
• Tone
• UDP
• USART Serial
• USB Keyboard
• USB Mouse
• WiFi Tester

Oh yes, then removing calls and #includes from files including “spark_wiring.h” that referenced particularly SPI, I2C and USART Serial. (This was probably where my gains came from.) Tedious? Yes.

For the record, I’m using the amazingly comprehensive PSPad as a developer GUI. I took a look at NetBeans’ 700MB install size, and said, “No way.” PSPad installs without any voliminous Java dependencies, and is only 15MB in size. Do I miss the polished professionalism of Particle IDE? Yes. Do I miss the ease of use? Sure. Worth it? Absolutely!

Trying to get rid of the compile errors the brute force way:

• Commented out a reference to “checks.mk” from “firmware-latest\build\common_tools.mk”–got rid of all those fatal: Not a git repository (or any of the parent directories): .git errors
• Completely stumped on how to get rid of the “stat not found” error. If I give it Stat.exe from MSYS 1.0, Stat crashes, and I get a stack dump. Not much of a solution…

Since switching to the LATEST branch on a local build (despite the continuing compiler issues), the output BIN (well, actually “.bin.pre_crc”) is 4,616 bytes smaller than the Particle Dev IDE’s output…and I haven’t removed anything from my code.
Everything works, with the sole exception of WiFi.ping(); It takes the two-second block, and returns 0. TCPClient can connect just fine to the same servername and communicate both ways, and WiFi.resolve() returns a valid IP address. So far, it appears that just WiFi.ping() is broken.

EDIT: The code for WiFi.ping() is practically the same between “LATEST” and “COMPILE-SERVER2”. @mdma, I see your name on the source code file . Two lines are different, namely uint32_t pingIPAddr = ip->ipv4; and system_tick_t lastTime = HAL_Timer_Get_Milli_Seconds(); stuff…but the timer stuff is working, as it blocks ~2 seconds each time. I don’t see why it isn’t working. But it isn’t.

[WebDust21]

Whew...finally found a solution on the forum for STAT.exe crashing! (I do feel all alone out here, though )

@bpr

Unless I'm mistaken, that looks like your USER path. C:\MinGW\msys\1.0\bin has to be at the very beginning of your SYSTEM path (NOT later in the user path).At least it did in my system.I'd want to see what is displayed when you type "path" without the quotes in the terminal

Bingo! No crash, no error, and I get an output that I don't quite understand:

   text       data        bss        dec        hex    filename
  97592        832      10076     108500      1a7d4    ../build/target/main/platform-0-lto/Project.elf

The BIN file is 98,428 bytes. But if the "DEC" value is to believed, I'm just about out of memory (according to @mdma, 110,592 bytes max.) But DFU-Util downloads the 98,428 bytes of the BIN file, not the 108,500 of the "stat" output.

WiFi.ping() is just simply, plainly broken. I don't understand how, as the changes don't seem major. Serial.println(WiFi.ping(WiFi.gatewayIP())); returns zero. Every time. I have confirmed that the IP address is correct in "wlan_hal.c", where the inet_ping function is found...and even tried reversing the sequence of the IP address bytes (3210 -> 0123). The function uses netapp_ping_send (found in "netapp.c"), which has not changed at all.

[peekay123]

@WebDust21, the “latest” branch is really for the Photon and it is early days for the new Core firmware. I have found that code which fit in flash in the 0.3.4 firmware environment no longer fits with the HAL code. The focus right now is really on the Photon and the Core update still needs lots of lovin’.