New library - Papertrail


#8

Hi @harrisonhjones, well…with the example program that @barakwei provided and if I put his library and code in my own small program, the Photon does have trouble connecting for some yet to be determined reason. If I remove the code from my program, it connects quickly, as normal.


#9

thanks @rickkas7! I
I have a version of the library with Electron support in a branch if anyone wants to make sure it actually works since I don’t have an Electron.
@ctmorrison - Hope you got it working, I have no idea how the library can cause that.


#10

Thanks for the awesome library. Just FYI, there is an outstanding request in the firmware repo for a Cellular.resolve() function. I’m going to see if I can get any movement on it.

Also, thinking about adding a section on “community log handlers” to the docs. Would you mind if I added yours?


#11

That’ll be great. I’m guessing we’ll need something similar for raspberry pi as well.

Feel free to mention the library in the docs or anywhere else.


#12

Thanks for the library! I’m having some trouble where not all my log statements are showing up, but pushing logs to PT is already a huge help with debugging.


#13

I’m glad you find it useful :smile:

I saw that after you lose connection to the cloud, you no longer receive events. I intend to fix this of course. I didn’t see random events not showing on papertrail. Which one do you see?


#14

I’m finding that log messages just never make it to Papertrail. For example, I have 3 log messages in my setup() function. On startup, the last one shows up in Papertrail about 25% of the time. It would be easier to debug if it never happened, but since it happens sometimes and not others, that makes it very strange.

I’m trying to troubleshoot some kind of freeze-up that happens in my firmware. I had hoped the UDP/Papertrail logging would help, and it has a little, but since I know that a lot of log messages are being lost, I can’t really trust it.

One thought I had was to cache the address that Wifi.resolve() gets back for the Papertrail host. I worry that DNS resolution might take a long time and/or cause UDP packets to back up. That’s just a wild guess though.


#15

Do you have SYSTEM_THREAD(ENABLED); in your code?


#16

The master branch in the git repo includes the IP address cache along with other stuff. I intend to upload this after some more testing, you can give it a try.
Also try to check to return value of m_udp.sendPacket.


#17

THANK YOU! @barakwei this is exactly what I need. I’m going to be including ParticlePapertrail in all of my projects. I’ll be sure to send you any thoughts/issues that arise for me, but so far so good.


#18

I am new to the platform. Thank you for this - a great step on Photon (with fw 0.6.1).

Unfortunately the same code does not run on Electron with 0.6.1, (and triggers a bug in the IDE, see below). The device breaths green or breaths cyan running the code below.

When breathing cyan, nothing gets sent to papertrail, the particle.variable is not registered, and the devices never starts blinking the blue LED. So the papertrail init seems blocking.

(Insert your own paper trail dem account credentials.)

// This #include statement was automatically added by the Particle IDE.
#include <papertrail.h>


// Slow 1fps blink blue board LED on 0.6.1 Photon and Electron
int led = D7; // blue board LED
int fps = 1000;

/*
SerialLogHandler logHandler(LOG_LEVEL_WARN, { // Logging level for non-application messages
    { "app", LOG_LEVEL_ALL } // Logging level for application messages
});
*/
//
PapertrailLogHandler papertailHandler("logsX.papertrailapp.com", XXXXX, "PapertrailSimpleDemo");

void setup() {
    Particle.variable("fps(ms)", fps);
    pinMode(led, OUTPUT);
    Log.trace("Starting, fps(ms): ", fps);
}

void loop() {
    digitalWrite(led, HIGH);
    delay(fps);
    digitalWrite(led, LOW);
    delay(fps);
}

It also triggers a wrong indication in the IDE:

When breathing cyan the device is listed as online in the console but variables does not show (“no variables registered”).

If I flash to the device breathing cyan via the IDE, the IDE indicates flashing is working fine but the device does not actually get flashed. Nothing shows up in the console log if active during the flashing.

So the device is probably online but can not be interacted with due to blocking code, so the “flash succesful” message is wrong:

followed by

This is on Mac OS 10.12.4 (16E195) with Safari Version 10.1 (12603.1.30.0.34)

I would appreciate to know if others have the lib running on Electron, thanks. It is a great debug option to have!


#19

Hi @thrmttnw
The git repo also has an issue on this matter. Unfortunately I’m still waiting for someone with an electron to debug this.

I’d really appreciate if anyone would help in this matter
Barak (still waiting for my Electron to arrive)


#20

I looked in to it briefly, adding the library code in two files to modify with the above example.

  • Replacing Cellular.resolve() with a manually resolved IP made no difference

  • Commenting out udp.begin() and udp.send() and just ret=1 in both cases, the rest of the code runs fine!

  • I tried to only do udp.begin() if particle.connected(), but it still blocks led blinking

It’s hard to send UDP packets without .begin() and I can’t go down that rabbit hole.

The change log for Electron states another difference between Photon and Electron:
https://docs.particle.io/reference/changelog/#udp-messaging

and “you can always switch protocols for your device in firmware” would be the next thing I would try, if I could find out how to do it …


#21

I also tried putting in delay(20) just before udp.begin(). Then the device connects, breaths cyan, and starts blinking the blue LED after downloading the binary, but a minute later or so, the device stops blinking the blue LED, still breathing cyan.

Pressing the reset button does not start LED blinking again. Only flashing the same binary again results in the blue LED blinking again, until about a minute later it stops.

So udp.begin seems to be causing a problem at system level.


#22

Seems like udp might be broken in 0.6.1

What prevents the library from running on 0.6.0?


#23

I’m pretty sure they broke the log handler in 0.6.1, so I had to fix it. You can you 0.6.0 with older version of the library.

I suggest to open a issue in the FW’s GitHub repo. I intend to do this once I find the time.

I extremely appreciate the investigation!! Many thanks!!


#24

@barakwei, great job on the library.

If you want a syslog client without using Papertrail, I just uploaded some sample code. I use the Kiwi syslog server and capture log messages from various programs. I have only tried compiling it on a Photon but I also have a Core and an Electron if anybody wants any additional versions.


#25

Hello @thrmttnw @syrinxtech @barakwei @rickkas7 Firmware 0.8.0-rc.3 Pre-release is out on the Web IDE for Core/Photon/P1/Electron. This release fixes the issue seen above where recursive logging would lock up the application thread on the Electron. It’s still not advisable to log too much over UDP on the Electron. I’ll suggest two that work well below. Please see the release notes on the Particle Firmware Updates Thread

// app level over papertrail
PapertrailLogHandler papertrailLogHandler1(PAPERTRAIL_HOSTNAME, PAPERTRAIL_PORT, "MyAppName", System.deviceID(),
    LOG_LEVEL_NONE, {
    { "app", LOG_LEVEL_ALL }
    // TOO MUCH!!! { "system", LOG_LEVEL_ALL },
    // TOO MUCH!!! { "comm", LOG_LEVEL_ALL }
});

// system info and app level over papertrail
PapertrailLogHandler papertrailLogHandler2(PAPERTRAIL_HOSTNAME, PAPERTRAIL_PORT, "MyAppName", System.deviceID(),
    LOG_LEVEL_NONE, {
    { "app", LOG_LEVEL_ALL },
    { "system", LOG_LEVEL_INFO }
    // MAYBE { "comm", LOG_LEVEL_INFO }
});

#26

Thanks @BDub those log levels are fair enough.

When logging is too much (ex. in combo with app functionality), how does it fail now and how can the device (be) recover(ed)?


#27

To be honest I haven’t seen it fail with the changes. I’ve also been using USB (LOG_LEVEL_ALL) and Cloud logging (see above) at the same time. I only caution against LOG_LEVEL_ALL because you can eat up your SIM data fast, especially if you forget to turn it off (or don’t add some kind of a timer to it :slight_smile: ). If you are not bothered by SIM data, give LOG_LEVEL_ALL a try and see how it does. Obviously the modem is going to be busy a lot so I wouldn’t be surprised if it appears to slow the ability to process the system thread (pub/sub/funcs/vars).