Photon Breathing Green, yet connected to Cloud


#1

I have a situation where my Photon is breathing Green, yet it is connected to the cloud (verified by refreshing the devices list in the Web IDE) and also seen in Particle Dev as online.

This Photon is connected to an MQTT broker. It starts off breathing cyan but after about 24 hours, it switches to breathing green and is no longer responding/receiving MQTT messages. I have Not set the SYSTEM_MODE in anyway. I also ensure I have a delay or Particle.process() is any tight loops.

What does the breathing green mean in this case and why/when would the Photon go into this state?


#2

Breathing green is doc’ed here:

https://docs.particle.io/guide/getting-started/modes/photon/

Here is what is says:

When the Photon is connected to a Wi-Fi network but not to the cloud, it will be breathing green.

This can be caused by the currently running application firmware which may interfere with the cloud maintenance tasks which are usually executed between iterations of loop() or via an explicit call of Particle.process(). That commonly happens when the code blocks for more than 10 seconds. In addition to regularly allowing for cloud maintenance (via dropping out of loop() and/or calling Particle.process()) you can take manual control of the connection, choose a better suited SYSTEM_MODE and/or opt for SYSTEM_THREAD(ENABLED). To correct the “offending” firmware you may need to flash new firmware either via USB or Safe Mode.

So are you sure that the MQTT library is non-blocking? Your code could be doing everything right but the library could be hurting you.


#3

Hi @bko,
Thank you for your reply.

Non-Blocking - In the context of TCP/IP, non-blocking would mean async I/O. Not it’s not async I/O it is synchronous. But I’ve used it with the Photon in many projects. And calling it’s loop() method within the main loop() method has never caused this issue.

What I’m doing differently (from my other projects), is that I’ve abstracted the connection (to the Broker) and reconnection logic into a different class. I’m still digging into this to figure out what’s really different. What has me confused is that both the Web IDE and Particle dev see this device as online (even after days and multiple refreshes). Also the green breathing always occurs only after approximately 24 hours. Never before that (like 1 hour or 2 hours etc.)


#4

What I meant by non-blocking is that your code does not block loop or somehow fails to call Particle.process() for more than 10 seconds. We have evidence (breathing green) that that is what is happening–the question is where is it happening? Is it happening in code that you yourself wrote or in code that you included from a library?


#5

Out of curiosity, what MQTT client/library are you running?


#6

The MQTT for Photon, Spark Core library


#7

@hirotakaster is on these forums… you might reach out to him.

You might also consider a spin up of an “out of the box version” on your photon without your recent class abstractions and see if the same results are replicated. I had a similar case where I was abstracting portions of the WolfSSL library. Peeling it back to original state would yield a go/no-go as far as your Photon state goes…

I’d also suggest exposing a particle.variable() to see if you can observe state changes in the Particle cloud when you cause a local change…


#8

Looking at the MQTT library, there is at least one data dependent do { } while loop that has the possibility of getting stuck if the data is not correct.

I would make a copy of the library and start instrumenting it with calls to do some debug logging and see what it is doing.

You could also try putting a time-out in the do/while loop so it can’t get stuck forever.


#9

Thanks for looking into this @bko. I was doing just that. I’ll start with doing some debug logging to see what’s going on or what it was doing when this occurs.


#10

if you okay, would you like to show your source code?


#11

@hirotakaster,

Yes, sure.

What I’ve found is that Serial.print statement is causing these problems. Simply removing the Serial.print statements makes the connection to the MQTT broker extremely stable. However, including Serial.print statements, makes the connection to the MQTT broker extremely unstable.

By unstable, what I mean is that the connection drops repeatedly. And then eventually after constantly connecting/disconnecting for about 24 hours, it goes into the breathing green mode.

If you have any corrections or guidance related to my class, I’m all ears.

Here is the .ino file

#include "NetworkConnectionManager.h"

unsigned long lastPublished;
/* Publish Temperature and Information every 5 seconds */
unsigned int publishInterval = 5000;

/* The MQTT Broker to use */
char *mqttBrokerUrl = "iot.eclipse.org";

/* The name of the Zone for this Device - 25 char max length */
const char *deviceZone = "Family Room";

/*
  commandTopic:
  This is the topic the Thermostat is a subscriber to. When you
  want to change the Settings of the Thermostat, you should
  publish messages to this topic
*/
const char *commandTopic = "test/therm/cmd";

/*
  eventTopic:
  This is the MQTT Topic that all Remote Sensors will publish
  their temperature and humidity information to. This is
  also the topic that the Thermostat will subscribe to in
  order to receive data from remote sensors. The thermost
  will publish its own sensor data to this topic as well
*/
const char *eventTopic = "test/therm/evt";

/*
  informationTopic:
  This is the MQTT Topic that the Thermostat will publish
  "information" to. This information, is essentially the
  Current Settings, such as the current Mode, current Zone
  and the Current Temperature it has been set to. In addition,
  the Thermostat also publishes data from all remote Sensors
  to this topic (all of this in the same message)
*/
const char *informationTopic = "test/therm/inf";

NetworkConnectionManager networkConnectionManager{
    mqttBrokerUrl, commandTopic, eventTopic, mqttEventCallback};

void setup() {
  // Serial.begin(9600);
  delay(5000);
}

void loop() {
  unsigned long now = millis();

  if (now - lastPublished > publishInterval) {
    publishTemperatureAndHumidity();
    publishInformation();
    lastPublished = millis();
  }
  networkConnectionManager.loop();
}

void publishTemperatureAndHumidity() {
  const char *buffer = "Z=Family Room&T=69.99&H=54.99";
  networkConnectionManager.publish(eventTopic, (uint8_t *)buffer,
                                   strlen(buffer), true, MQTT::EMQTT_QOS::QOS1);
  networkConnectionManager.flashLed(255, 102, 0, 2);
}

void publishInformation() {
  const char *buffer = "AM=Off&AZ=Master Bedroom&ST=68.00&AO=false|N=Master "
                       "Bedroom&T=69.48&H=55.22|N=Family Room&T=69.84&H=57.96";
  networkConnectionManager.publish(informationTopic, (uint8_t *)buffer,
                                   strlen(buffer), true, MQTT::EMQTT_QOS::QOS1);
}

void mqttEventCallback(char *topic, byte *payload, unsigned int length) {}

NetworkConnectionManager.h

/*
  NetworkManager.h - This class abstracts the management
  of the Wifi and Mqtt Clients. Besides connecting to
  both, it also ensure reconnecting if required
  This class is Particle Photon Specific
  Created by Shiv Kumar, 2015.
  website: http://www.matlus.com
  Released into the public domain.
*/
#ifndef NetworkConnectionManager_h
#define NetworkConnectionManager_h
#include <MQTT.h>

class NetworkConnectionManager {
public:
  NetworkConnectionManager(char *mqttBrokerUrl,
                           const char *commandSubscribeTopic,
                           const char *eventSubscribeTopic,
                           void (*messageReceivedCallback)(char *, uint8_t *,
                                                           unsigned int));
  ~NetworkConnectionManager();
  void loop();
  void ensureMqttBrokerConnectivity();
  bool publish(const char *topic, const uint8_t *payload, unsigned int plength,
               bool retain, MQTT::EMQTT_QOS qos);
  void flashLed(uint8_t red, uint8_t green, uint8_t blue, uint8_t count);

private:
  std::unique_ptr<MQTT> m_mqttClient = nullptr;
  char *m_mqttBrokerUrl = nullptr;
  const char *m_commandSubscribeTopic = nullptr;
  const char *m_eventSubscribeTopic = nullptr;
  int m_ConnectionAttempts = 0;
  bool m_canPublish;
  void ensureWiFiConnectivity();
  void resolveMqttBrokerAddress(const char *mqttBrokerUrl);
  bool tryConnectToMqttBroker(uint8_t noOfRetries);
  void breathingLed(uint8_t red, uint8_t green, uint8_t blue);
};

#endif

NetworkConnectionManager.cpp

#include "NetworkConnectionManager.h"

NetworkConnectionManager::NetworkConnectionManager(
    char *mqttBrokerUrl, const char *commandSubscribeTopic,
    const char *eventSubscribeTopic,
    void (*messageReceivedCallback)(char *, uint8_t *, unsigned int))
    : m_mqttBrokerUrl(mqttBrokerUrl),
      m_commandSubscribeTopic(commandSubscribeTopic),
      m_eventSubscribeTopic(eventSubscribeTopic),
      m_mqttClient(std::unique_ptr<MQTT>(
          new MQTT(mqttBrokerUrl, 1883, messageReceivedCallback))) {}

NetworkConnectionManager::~NetworkConnectionManager() {}

void NetworkConnectionManager::loop() {
  m_canPublish = false;

  if (!m_mqttClient->loop()) {
    ensureMqttBrokerConnectivity();
  }

  m_canPublish = true;
}

bool NetworkConnectionManager::publish(const char *topic,
                                       const uint8_t *payload,
                                       unsigned int plength, bool retain,
                                       MQTT::EMQTT_QOS qos) {
  if (m_canPublish) {
    return m_mqttClient->publish(topic, payload, plength, retain, qos);
  }
}

void NetworkConnectionManager::ensureMqttBrokerConnectivity() {
  if (m_mqttClient->isConnected()) {
    return;
  }

  ensureWiFiConnectivity();
  resolveMqttBrokerAddress(m_mqttBrokerUrl);
  bool connected = tryConnectToMqttBroker(10);

  if (!connected) {
    // Serial.println("MQTT Broker Connection Re-Try attempts exceeded...");
    // Serial.println("System Resetting");
    System.reset();
  }
}

void NetworkConnectionManager::ensureWiFiConnectivity() {
  while (!WiFi.ready()) {
    Particle.process();
    // Serial.println("Connecting to WiFi...");
    flashLed(255, 51, 204, 3);
    delay(1000);
  }

  // Serial.println("Connected to WiFi");
  // Serial.print("IP Address: ");
  // Serial.println(WiFi.localIP());
}

void NetworkConnectionManager::resolveMqttBrokerAddress(
    const char *mqttBrokerUrl) {
  // Serial.print("Resolving MQTT Broker Address: ");
  // Serial.print(mqttBrokerUrl);
  uint8_t resolveRetryCount = 0;
  IPAddress ip;
  while (!ip) {
    Particle.process();
    // Serial.print(".");
    ip = WiFi.resolve(mqttBrokerUrl);
    resolveRetryCount++;
    if (resolveRetryCount == 10) {
      // Serial.println("Unable to Resolve MQTT Broker Address");
      // Serial.println("System Resetting");
      delay(3000);
      System.reset();
    }
    delay(1000);
  }
  // Serial.println("\r\nMQTT Broker Address resolved");
}

bool NetworkConnectionManager::tryConnectToMqttBroker(uint8_t noOfRetries) {
  uint8_t noOfRetryAttepmts = 0;

  while (!m_mqttClient->isConnected() && noOfRetryAttepmts < noOfRetries) {
    // Serial.println("\r\nMQTT Client Not Connected");

    byte mac[6];
    WiFi.macAddress(mac);
    char clientId[18];
    sprintf(clientId, "%02x%02x%02x%02x%02x%02x%02x", mac[0], mac[1], mac[2],
            mac[3], mac[4], mac[5], random(0xffff));
    // Serial.print("ClientId: ");
    // Serial.println(clientId);

    // Serial.print("Connecting to MQTT Broker: ");
    // Serial.println(m_mqttBrokerUrl);

    m_mqttClient->connect(clientId);

    if (m_mqttClient->isConnected()) {
      // Serial.println("\r\nConnected to MQTT Broker");
      m_mqttClient->subscribe(m_commandSubscribeTopic);
      m_mqttClient->subscribe(m_eventSubscribeTopic);

      // Serial.print("Subscribed to Command Topic: ");
      // Serial.println(m_commandSubscribeTopic);
      // Serial.print("Subscribed to Event Topic: ");
      // Serial.println(m_eventSubscribeTopic);
      // Serial.println("");
    } else {
      // Serial.print("Failed to connect to MQTT Broker: ");
      // Serial.println(noOfRetryAttepmts + 1);
      // Serial.println("Creating a new instance of Mqtt Client");
      // m_mqttClient = createMqttClient();
      // Serial.println("Trying again in 1 second...");
      flashLed(255, 51, 204, 5);
      delay(1000);
      noOfRetryAttepmts++;
    }
  }

  return (noOfRetryAttepmts < noOfRetries);
}

void NetworkConnectionManager::flashLed(uint8_t red, uint8_t green,
                                        uint8_t blue, uint8_t count) {
  RGB.control(true);

  RGB.color(red, green, blue);
  for (uint8_t i = 0; i < count; i++) {
    RGB.brightness(255);
    delay(50);
    RGB.brightness(64);
    delay(50);
  }

  RGB.control(false);
}

void NetworkConnectionManager::breathingLed(uint8_t red, uint8_t green,
                                            uint8_t blue) {
  RGB.control(true);

  RGB.color(red, green, blue);

  for (uint8_t i = 0; i < 254; i = i + 10) {
    RGB.brightness(i);
    delay(100);
    Particle.process();
  }

  for (uint8_t i = 254; i > 0; i = i - 10) {
    RGB.brightness(i);
    delay(100);
    Particle.process();
  }

  RGB.control(false);
}

I also have a question related to your code in the MQTT class.

MQTT.cpp

void MQTT::initialize(char* domain, uint8_t *ip, uint16_t port, int keepalive, void (*callback)(char*,uint8_t*,unsigned int), int maxpacketsize) {
    this->callback = callback;
    this->qoscallback = NULL;
    if (ip != NULL)
        this->ip = ip;
    if (domain != NULL)
        this->domain = domain;
    this->port = port;
    this->keepalive = keepalive;
    // if maxpacketsize is over MQTT_MAX_PACKET_SIZE.
    this->maxpacketsize = (maxpacketsize <= MQTT_MAX_PACKET_SIZE ? MQTT_MAX_PACKET_SIZE : maxpacketsize);
    buffer = new uint8_t[this->maxpacketsize];
    this->_client = new TCPClient();
}

In the above code, you create a TCPClient object. But in the destructor, you don’t delete that object. Isn’t that a memory leak, since _client member is declared like so?:
TCPClient *_client;
Mind you, I’m by no means a seasoned C++ programmer.


#12

Thank you @shiv . your point is exactly!! This pointer was used for compatibility with Arduino and others platforms, but now don’t use.
now I update 0.4.23@MQTT. please check it.


#13

Great, thank you!