Local Compiler Error that adds comma to my code, fails

justicefreed_amper · January 9, 2019, 10:27pm

The cloud compiler and calling particle compile electron --target {target} both haven’t been working for me for a while (Unknown Compiler Error). However the local compiler toolchain via particle workbench has been working great, until now.

The local compiler toolchain creates a copy of the ino file, and renames it into a cpp file, as part of the preprocessing.

I keep getting this error:

/Users/justicereed/Documents/Engineering/Hardware/gateway-firmware/src/gateway-firmware.ino:1713:23: error: expected primary-expression before ',' token
                         {
                       ^
/Users/justicereed/Documents/Engineering/Hardware/gateway-firmware/src/gateway-firmware.ino:1713:26: error: expected primary-expression before '{' token
                         {

Here is my code in that section:

                    if (sd_rd_char_payload.payload[14] == 't')
                    {
                        if (cur_debug_request.inc_debug)
                        {
                            log(LOG_LVL_TRACE, "Putting log trace msg into pub queue");
                            sd_to_char_pub.put(sd_rd_char_payload);
                        }
                    }

And here is the code in the automatically created .cpp file:

                    if (sd_rd_char_payload.payload[14] == 't')
                    {
                        if (cur_debug_request.inc_debug)
                      ,  {
                            log(LOG_LVL_TRACE, "Putting log trace msg into pub queue");
                            sd_to_char_pub.put(sd_rd_char_payload);
                        }
                    }

Notice the ghost comma has been added seemingly by the preprocessor! The comma is always added exactly at line 1747, position 22.

Anyone know what on earth could cause this?

justicefreed_amper · January 9, 2019, 10:43pm

Hmm, before when I deleted the .cpp the issue had continued to present, but this time when I deleted it the issue hasn’t come back. Not sure why it started working again. Not sure whether or not to mark this as resolved, as there definitely was a problem that resolved itself without any intentional change on my part.

EDIT: now it’s back on the same line, though at position 7

m_m · January 9, 2019, 11:15pm

@justicefreed_amper

ooof! that’s no good. can you run Particle: Audit Environment and share the non-sensitive bits here? also, if you are comfortable sharing, i’d love to see if i can repro it over here using your original source (DM me a zip if so).

re: your problems running the cloud compiler, i wonder if it’s doing the same thing (adding a comma)? i assume you’ve scanned the response log it returns (assuming that’s the Unknown Compiler Error bit)? if you run the Particle: Cloud Compile command, remember to click “view log” in the error notification that pops up.

justicefreed_amper · January 9, 2019, 11:21pm

Particle: Audit Environment Output:

*** WARNING: MAY CONTAIN CONFIDENTIAL INFORMATION - PLEASE SANITIZE BEFORE SHARING ***
To report an issue, visit:
https://community.particle.io/t/information-how-to-report-bugs-and-provide-feedback/44245

{
    "username": "{REDACTED}",
    "workspace": {
        "name": "gateway-firmware (Workspace)",
        "isWorkspace": true,
        "configFile": "gateway-firmware.code-workspace",
        "root": "/Users/justicereed/Documents/Engineering/Hardware/gateway-firmware",
        "files": [
            ".DS_Store",
            ".git",
            ".git/COMMIT_EDITMSG",
            ".git/FETCH_HEAD",
            ".git/HEAD",
            ".git/ORIG_HEAD",
            ".git/config",
            ".git/description",
            ".git/hooks",
            ".git/index",
            ".git/info",
            ".git/logs",
            ".git/modules",
            ".git/objects",
            ".git/packed-refs",
            ".git/refs",
            ".git/sequencer",
            ".gitignore",
            ".vscode",
            ".vscode/settings.json",
            "README.md",
            "gateway-firmware.code-workspace",
            "{REDACTED}",
            "lib",
            "lib/.DS_Store",
            "{REDACTED}",
            "project.properties",
            "src",
            "{REDACTED}",
            "src/gateway-firmware.cpp",
            "src/gateway-firmware.ino",
            "{REDACTED}",
            "target",
            "target/gateway-firmware.bin",
            "target/gateway-firmware.bin.crc_block",
            "target/gateway-firmware.bin.no_crc",
            "target/gateway-firmware.elf",
            "target/gateway-firmware.hex",
            "target/gateway-firmware.lst",
            "target/gateway-firmware.map",
            "target/obj"
        ]
    },
    "vscode": {
        "appName": "Visual Studio Code",
        "appRoot": "/Applications/Visual Studio Code.app/Contents/Resources/app",
        "machineId": "4cc4cb9a12ea10c756da5e4376e9dea6d7c38af8833f85573107faa9b6beb4e8",
        "sessionId": "c545e37b-f9a9-405f-a4a7-131567e2b1f31547070536618",
        "version": "1.30.2"
    },
    "cli": {
        "binpath": "/Users/justicereed/.vscode/extensions/particle.particle-vscode-core-1.0.0-alpha.6/src/cli/bin/darwin/amd64/particle",
        "ok": true,
        "version": "1.36.3"
    },
    "localCompiler": {
        "dependencies": [
            "deviceOS@0.6.4",
            "gcc-arm@4.9.3",
            "buildtools@1.0.1",
            "buildscripts@1.0.0"
        ]
    },
    "platform": {
        "os": "darwin",
        "type": "Darwin",
        "release": "18.0.0",
        "arch": "x64",
        "path": {
            "key": "PATH",
            "value": "/Users/justicereed/.nvm/versions/node/v8.9.4/bin:/Users/justicereed/Library/Python/3.6/bin:/Users/justicereed/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Postgres.app/Contents/Versions/latest/bin"
        },
        "homeDir": "/Users/justicereed",
        "particleDir": "/Users/justicereed/.particle",
        "localCompilerToolchainDir": "/Users/justicereed/.particle/toolchains"
    },
    "env": {
        "TMPDIR": "/var/folders/60/_39c6nf109v28rbbvzcwwxnh0000gn/T/",
        "__CF_USER_TEXT_ENCODING": "0x1F5:0x0:0x0",
        "SHELL": "/bin/bash",
        "HOME": "/Users/justicereed",
        "Apple_PubSub_Socket_Render": "/private/tmp/com.apple.launchd.d5yGv5hPUK/Render",
        "SSH_AUTH_SOCK": "/private/tmp/com.apple.launchd.CEQc881I3b/Listeners",
        "PATH": "/Users/justicereed/.nvm/versions/node/v8.9.4/bin:/Users/justicereed/Library/Python/3.6/bin:/Users/justicereed/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Postgres.app/Contents/Versions/latest/bin",
        "LOGNAME": "justicereed",
        "XPC_SERVICE_NAME": "com.microsoft.VSCode.1508",
        "USER": "justicereed",
        "XPC_FLAGS": "0x0",
        "VSCODE_NLS_CONFIG": "{\"locale\":\"en-us\",\"availableLanguages\":{},\"_languagePackSupport\":true}",
        "VSCODE_NODE_CACHED_DATA_DIR": "/Users/justicereed/Library/Application Support/Code/CachedData/61122f88f0bf01e2ac16bdb9e1bc4571755f5bd8",
        "VSCODE_LOGS": "/Users/justicereed/Library/Application Support/Code/logs/20190109T154855",
        "VSCODE_IPC_HOOK": "/Users/justicereed/Library/Application Support/Code/1.30.2-main.sock",
        "VSCODE_PID": "13903",
        "NVM_BIN": "/Users/justicereed/.nvm/versions/node/v8.9.4/bin",
        "NVM_CD_FLAGS": "",
        "NVM_DIR": "/Users/justicereed/.nvm",
        "PWD": "/",
        "SHLVL": "1",
        "_": "/Applications/Visual Studio Code.app/Contents/MacOS/Electron",
        "AMD_ENTRYPOINT": "vs/workbench/node/extensionHostProcess",
        "PIPE_LOGGING": "true",
        "VERBOSE_LOGGING": "true",
        "VSCODE_IPC_HOOK_EXTHOST": "/var/folders/60/_39c6nf109v28rbbvzcwwxnh0000gn/T/vscode-ipc-0f6abdc8-516b-4717-b868-af2baa0fa80c.sock",
        "VSCODE_HANDLES_UNCAUGHT_ERRORS": "true",
        "VSCODE_LOG_STACK": "false",
        "VSCODE_LOG_LEVEL": "undefined",
        "VSCODE_PREVENT_FOREIGN_INSPECT": "true",
        "CRASH_REPORTER_START_OPTIONS": "{\"companyName\":\"Microsoft\",\"productName\":\"VSCode\",\"submitURL\":\"https://rink.hockeyapp.net/api/2/apps/21a48a66799e47fea4f52c0ff81e803d/crashes/upload\",\"extra\":{\"vscode_version\":\"1.30.2\",\"vscode_commit\":\"61122f88f0bf01e2ac16bdb9e1bc4571755f5bd8\",\"vscode_sessionId\":\"c545e37b-f9a9-405f-a4a7-131567e2b1f31547070536618\",\"processName\":\"extensionHost\"},\"crashesDirectory\":\"/var/folders/60/_39c6nf109v28rbbvzcwwxnh0000gn/T\"}",
        "ELECTRON_RUN_AS_NODE": "1",
        "APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL": "true"
    },
    "versions": {
        "http_parser": "2.7.0",
        "node": "8.9.3",
        "v8": "6.1.534.41",
        "uv": "1.15.0",
        "zlib": "1.2.11",
        "ares": "1.10.1-DEV",
        "modules": "57",
        "nghttp2": "1.25.0",
        "openssl": "1.0.2n"
    },
    "extensions": [
        {
            "id": "ms-vscode.node-debug",
            "version": "1.30.4"
        },
        {
            "id": "ms-vscode.node-debug2",
            "version": "1.30.3"
        },
        {
            "id": "ms-vscode.references-view",
            "version": "0.0.18"
        },
        {
            "id": "marus25.cortex-debug",
            "version": "0.1.21"
        },
        {
            "id": "ms-vscode.cpptools",
            "version": "0.20.1"
        },
        {
            "id": "particle.particle-vscode-core",
            "version": "1.0.0-alpha.6"
        },
        {
            "id": "particle.particle-vscode-snippets",
            "version": "1.0.0-alpha.6"
        },
        {
            "id": "particle.particle-vscode-theme",
            "version": "1.0.0-alpha.6"
        }
    ]
}

*** WARNING: MAY CONTAIN CONFIDENTIAL INFORMATION - PLEASE SANITIZE BEFORE SHARING ***
To report an issue, visit:
https://community.particle.io/t/information-how-to-report-bugs-and-provide-feedback/44245

Regarding the normal compilation process, I believe it to be a separate issue. I ran into it before, and it seemed to be a problem with not being able to properly link library header files. I remember fixing it months ago, but of course not how I did!

Here’s the output from the verbose compilation for particle compile:

bash-3.2$ particle compile electron --target 0.6.4 -v

Compiling code for electron
Targeting version: 0.6.4

Including:
    {REDACTED}
    project.properties
attempting to compile firmware

undefined
Compile failed: Compiler encountered an error
VError: Compile failed: Compiler encountered an error
    at /Users/justicereed/.particle/node_modules/particle-cli/dist/cmd/cloud.js:320:10
    at process._tickCallback (node.js:401:9)
caused by: Error: Compiler encountered an error
    at /Users/justicereed/.particle/node_modules/particle-cli/dist/cmd/cloud.js:336:11
    at process._tickCallback (node.js:401:9)

m_m · January 9, 2019, 11:29pm

could you run a cloud compile right now? cmd+shift+p then pick Particle: Cloud Compile

justicefreed_amper · January 9, 2019, 11:30pm

Just did, output:


Compiling code for electron
Targeting version: 0.6.4

Including:
   {REDACTED}
    project.properties
attempting to compile firmware 

undefined
Compile failed: Compiler encountered an error

	at makeError (/Users/justicereed/.vscode/extensions/particle.particle-vscode-core-1.0.0-alpha.6/node_modules/execa/index.js:174:9)
	at module.exports.Promise.all.then.arr (/Users/justicereed/.vscode/extensions/particle.particle-vscode-core-1.0.0-alpha.6/node_modules/execa/index.js:278:16)
	at <anonymous>

- - - -

m_m · January 9, 2019, 11:45pm

a big ask, i realize but is there any chance you could write a failing test over here https://github.com/particle-iot/wiring-preprocessor

that’s what both the local and cloud compilers use under the hood.

edit: this is an example: https://github.com/particle-iot/wiring-preprocessor/tree/master/test/fixtures/long_ino

justicefreed_amper · January 9, 2019, 11:51pm

I wouldn’t be able to upload the actual code currently causing issues because of proprietary considerations, but I can see if I could somehow reproduce it with something else.

At a minimum though I could at least run the tests locally with my code and see if anything is to be gleaned from that. I’ll give that a shot right now, and if that isn’t helpful / conclusive I’ll do my best to come up with a generic fixture for the condition I can pass around.

m_m · January 9, 2019, 11:53pm

that’d be super-helpful thanks so much

justicefreed_amper · January 9, 2019, 11:58pm

EDIT: Nevermind, the test passes - I hadn’t edited the #line 22 entry to the right filepath. Back to square one - I’ll keep trying to reproduce the issue in the test when it fails on my main build

justicefreed_amper · January 10, 2019, 12:12am

So I have a preliminary failing test. It seems to be auto reducing some extra whitespace that doesn’t appear in the preprocessed .cpp file I copied the expected result from. Doesn’t seem expected, what do you think? I can try and edit the files to match the expected to see what else happens if that sounds helpful.


> wiring-preprocessor@1.0.1 test /Users/justicereed/Documents/Engineering/Hardware/wiring-preprocessor
> mocha test/ -R spec



  ✓ processes application h after includes
  ✓ processes commented includes
  ✓ processes commented no preprocessor
  ✓ processes custom types
  ✓ processes example app
  ✓ processes ifdef
  ✓ processes ifelse
  ✓ processes includes application h
  ✓ processes includes particle h
  ✓ processes long ino
  1) processes no comma inject
  ✓ processes other includes
  ✓ processes particle no preprocessor
  ✓ processes spark no preprocessor
  ✓ processes typedef types
  Wiring preprocessor
    ✓ loads the processor


  15 passing (46s)
  1 failing

  1) processes no comma inject:

      AssertionError: expected '{REDACTED}'
      + expected - actual

       //
       //------------------------------------------------------------------------------
       //------------------------------------------------------------------------------
       
      -bool waitCloudDisconnect()	{	return !Particle.connected();	}
      +bool waitCloudDisconnect()  {   return !Particle.connected();   }
       
       bool waitCellDisconnect()
       {
           #if Wiring_Cellular
      -    	return !Cellular.ready();
      +        return !Cellular.ready();
           #elif Wiring_WiFi
               return !WiFi.ready();
           #endif
       }
       
      -bool waitCellOff() 			{	return !network_status_local;	}
      +bool waitCellOff()          {   return !network_status_local;   }
       
      -bool waitCellOn() 			{	return network_status_local;	}
      +bool waitCellOn()           {   return network_status_local;    }
       
       //..............................................................................
       /*
           checkConnection() is called by loop() on the main application thread
--
                           Particle.disconnect();
                           delay(100);
                           log(LOG_LVL_INFO, "Resetting Modem...");
                           #if Wiring_Cellular
      -                		// 16:MT silent reset (with detach from network and saving of NVM parameters), with reset of the SIM card
      -                		Cellular.command(30000, "AT+CFUN=16\r\n");
      +                        // 16:MT silent reset (with detach from network and saving of NVM parameters), with reset of the SIM card
      +                        Cellular.command(30000, "AT+CFUN=16\r\n");
                               delay(100);
      -                		Cellular.off();
      +                        Cellular.off();
                           #elif Wiring_WiFi
                               WiFi.off();
                           #endif
      -            		delay(200);
      +                    delay(200);
                       }
                       // Now let's reset the electron
                       log(LOG_LVL_INFO, "Resetting Device...");
                       reset_handler();
      
      at Promise.all.then (test/index.spec.js:47:29)
      at <anonymous>



npm ERR! Test failed.  See above for more details.

m_m · January 10, 2019, 12:16am

i was hoping you’d be able to repro the comma injection. on the face of it, whitespace changes don’t jump out as a problem. honestly stumped at this point.

justicefreed_amper · January 10, 2019, 12:19am

I would agree, though I find it strange that “.cpp A” generated by the preprocessor doesn’t match the test “.cpp B” also generated by the preprocessor.

I’ll try and get rid of the whitespace changes and see if I can get it to either pass or catch the comma.

justicefreed_amper · January 10, 2019, 12:44am

Once I fixed the whitespace stuff, the test passes. I’ve tested a few different versions of my firmware that fail using the local compiler but pass the test. Makes me wonder if it’s external to the preprocessor somehow? There isn’t anything else that I can really see in the process flow that would be adding a character. Also really weird that it shows up on the same line number regardless of the code at that line or around it.

Worth noting about how I run the local compiler around this issue:

I run make -f $PARTICLE_MAKEFILE compile-user and it fails with an error driven by the comma. This also creates the .cpp file.
I delete the comma character from the .cpp file by hand and save.
I re-run make -f $PARTICLE_MAKEFILE compile-user, and process succeeds all the way, giving me my compiled binary.

So I guess the preprocessor sees that my files haven’t changed, so it doesn’t preprocess again, but then the rest of the compiler still gets called?

Edit: also btw this time the comma got put into a different line. I edited some code up higher in my program so there seems to be a trigger somewhere above it. I’ll keep messing around and see if I can narrow it down.

Edit2: If I add and remove lines of code (eg declare like 5 ints), the line number with the comma changes, but not by the number of lines I added…

m_m · January 10, 2019, 1:37am

Makes me wonder if it’s external to the preprocessor somehow?

yeah, that’s a good theory. we do run the preprocessor below a couple of other layers - first is our Makefile (see: ~/.particle/toolchains/buildscripts/1.0.1/Makefile) which then calls out to the CLI to preprocess files.

so that’s another thing to try: run the Particle: Launch CLI command, then particle preprocess <path/to/your.ino> in the terminal that launches and see if it adds the comma there. that should help narrow things down a bit.

I run make -f $PARTICLE_MAKEFILE compile-user

are you running via the Particle: Launch Compiler Shell command? have you tried the plain local compilation tasks? from VSCode’s main menu, select “Terminal > Run Tasks…”, then run Particle: Compile application (local). it’s running the same code under the hood so i would be surprised if you saw different behavior but

oh yeah, just for sanity, remember to run the Clean (e.g. Particle: Clean application (local)) before just to be extra-cautious (really shouldn’t matter though).

above all, thanks for reporting and troubleshooting the issue with me - it’s super-helpful even if we don’t find the smoking gun right away

jvanier · January 10, 2019, 2:12pm

There seems to be some back and forth over what the error is. Is the current theory still that something is being weird with the preprocessor? It’s quite possible since it’s a bunch of clever regular expressions that parses the ino file and inserts header includes and function declarations.

Here are a couple things you can try:

Add #pragma PARTICLE_NO_PREPROCESSOR at the top of your ino file then the only change between the ino and cpp file should be that the #pragma line gets commented. See if the comma disappears in this case. You’ll probably get additional compiler errors since your ino file will be missing the declarations that would have been added by the preprocessor
You can run the preprocessor step by itself to quickly iterate on a smaller file that reproduces the issue by doing particle preprocess gateway-firmware.ino

This is the case. The preprocessor will only run in Workbench if the cpp file is missing or the ino file changed (standard Makefile semantics).

justicefreed_amper · January 10, 2019, 3:36pm

Great suggestions!

@m_m

run the Clean (e.g. Particle: Clean application (local))

I have been running make -f $PARTICLE_MAKEFILE clean-[user, all] in the midst of all this, which I assume is the same underlying call, but I ran this as a task to be certain.

from VSCode’s main menu, select “Terminal > Run Tasks…”, then run Particle: Compile application (local)

Same issue.

@jvanier

Add #pragma PARTICLE_NO_PREPROCESSOR at the top of your ino file

When running this, I get expected errors from the lack of preprocessing, but when inspecting the .cpp file no comma seems to be present. Which would in theory say that the preprocessor is causing the specific issue.

run the preprocessor step by itself to quickly iterate on a smaller file that reproduces the issue by doing particle preprocess gateway-firmware.ino

If I run this with the #pragma above there is no comma. If I run without, as normal, the comma appears. So somehow there appears to be a difference between what my preprocessor is doing and what the preprocessor is doing as called in the test fixture.

I had previously wondered if I just had a corrupted file somehow, so I’ve already tried uninstalling and reinstalling the toolchain (all the ones I had installed, only reinstalled the one I was trying to use). But on deeper reflection it appears that the preprocessor itself isn’t tied to the firmware-specific toolchain, and that I would have to reinstall the Particle CLI. The environment var referenced in the makefile for the Particle CLI used to call the preprocessor is the workbench specific one, (the “Particle Workbench” extension). After uninstalling, redownloading, and reinstalling the extension I get the same error.

And of course after doing that I realized that running particle preprocess I was using my standard cli installation anyways, so both preprocessors packaged into the cli behave identically for me (with the comma injection), while simultaneously passing the test fixture preprocessor with no issues.

So I guess my next thought is: what is the difference between the way the wiring-preprocessor as called and run from the repo and the way the preprocessor is called and ran from the cli, given the constraint that the #pragma PARTICLE_NO_PREPROCESSOR seems to remove the part of the process that injects the comma.

EDIT: and keep in mind that the line number where the comma appears does seem to change in some cases dependent on how I change my code, but not in a linear way (though it is the same each time if I don’t add or remove lines of code). I’ll keep messing with that when I have some spare time today just in case I can better narrow down how this behaves.

justicefreed_amper · January 10, 2019, 4:15pm

So I tried a fun little thing.

I installed wiring-preprocessor and fs in my project directory, and wrote the following js file:

const fs = require('fs');
const preprocessor = require('wiring-preprocessor');

const inoFile = fs.readFileSync('src/gateway-firmware.ino', 'utf8');
const cppFile = preprocessor.processFile('src/gateway-firmware.ino', inoFile);
fs.writeFileSync('src/gateway-firmware.cpp', cppFile);

I edited the $PARTICLE_MAKEFILE makefile, and instead of calling the Particle CLI preprocessor, I explicitly called my own preprocess.js, above. Then the entire compilation ran perfectly.

It seems to pretty clearly indicate that there is no difference except for the way the preprocessor is called and the actual preprocessor used. Both seem to be using v1.0.1 of wiring-preprocessor.

I’ll try and further edit my js file to match the exact way the particle preprocessor command opens and edits the files and reply with the results

justicefreed_amper · January 10, 2019, 4:27pm

So I copied the file cmd/preprocess.js from the cli into my own local file ./preprocess_cmd.js, and then re-ran the compiler using the exact same method as the cli does changing my preprocess.js file to be:

const PreprocessCommand = require('./preprocess_cmd');
console.log("Running preprocessor on file: %s", process.argv);
return new PreprocessCommand().preprocess(process.argv[2]);

Now the comma appears again! Keep in mind that this is using the exact same install of wiring-preprocessor that I used in my successful test above. So it seems the issue has been narrowed down to the cli file src/cmd/preprocess.js. I’ll see if I can further debug within that file.

m_m · January 10, 2019, 4:36pm

Now the comma appears again!

does it reproduce 100% of the time?

So it seems the issue has been narrowed down to the cli file src/cmd/preprocess.js. I’ll see if I can further debug within that file.

thanks i assume you’ve found the source repo for the cli by now but just in case, it’s over here https://github.com/particle-iot/particle-cli