Problem with porting Exosite Arduino library

I tried to port Exosite library but I have a problem!
The library works somewhat but I do not know if the problem is the firmware or do I mistake somewhere.
This is the older version of the library. With the newer version I did not have success. A newer version is located at the following link: https://github.com/exosite-garage/arduino_exosite_library

The next problem occurs: between two consecutive sending variable appears delay of about 8 to 10 seconds!
Delay also appears with the send only one variable in the loop.
It is not problem when using Arduino Uno and Ethernet shield.

Code is attached:

#define serverName        "m2.exosite.com"

#define MAXTIMINGS 85
 
#define cli noInterrupts
#define sei interrupts
 
#define DHT11 11
#define DHT22 22
#define DHT21 21
#define AM2301 21
 
#define NAN 999999

    TCPClient client;
    String cik = "00000000000000000000000000000000000000"; <-- Fill in your CIK here!
    char rxdata[150];
    int ret;
    int stringPos;
    bool DataRx;
    bool RxLoop;
    char c;
    unsigned long timeout_time;
    unsigned long time_now;
    unsigned long timeout;
    String myDataString;

int sendToCloud(String res, int value);
int readFromCloud(String res ,String* pResult);

class DHT {
    private:
        uint8_t data[6];
        uint8_t _pin, _type, _count;
        bool read(void);
        unsigned long _lastreadtime;
        bool firstreading;
    
    public:
        DHT(uint8_t pin, uint8_t type, uint8_t count=6);
        void begin(void);
        float readTemperature(bool S=false);
        float convertCtoF(float);
        float readHumidity(void);
 
};

DHT::DHT(uint8_t pin, uint8_t type, uint8_t count) {
    _pin = pin;
    _type = type;
    _count = count;
    firstreading = true;
}
 
 
void DHT::begin(void) {
    // set up the pins!
    pinMode(_pin, INPUT);
    digitalWrite(_pin, HIGH);
    _lastreadtime = 0;
}
 
//boolean S == Scale.  True == Farenheit; False == Celcius
float DHT::readTemperature(bool S) {
    float _f;
    
    if (read()) {
        switch (_type) {
            case DHT11:
                _f = data[2];
                
                if(S)
                    _f = convertCtoF(_f);
                
                return _f;
                
                
            case DHT22:
            case DHT21:
                _f = data[2] & 0x7F;
                _f *= 256;
                _f += data[3];
                _f /= 10;
                
                if (data[2] & 0x80)
                    _f *= -1;
                    
                if(S)
                    _f = convertCtoF(_f);
                
                return _f;
        }
    }
    
    return NAN;
}

float DHT::convertCtoF(float c) {
    return c * 9 / 5 + 32;
}
 
 
float DHT::readHumidity(void) {
    float _f;
    if (read()) {
        switch (_type) {
            case DHT11:
                _f = data[0];
                return _f;
                
            case DHT22:
            case DHT21:
                _f = data[0];
                _f *= 256;
                _f += data[1];
                _f /= 10;
                return _f;
        }
    }
    
    return NAN;
}

bool DHT::read(void) {
    uint8_t laststate = HIGH;
    uint8_t counter = 0;
    uint8_t j = 0, i;
    unsigned long currenttime;
    
    // pull the pin high and wait 250 milliseconds
    digitalWrite(_pin, HIGH);
    delay(250);
    
    currenttime = millis();
    if (currenttime < _lastreadtime) {
        // ie there was a rollover
        _lastreadtime = 0;
    }
    
    if (!firstreading && ((currenttime - _lastreadtime) < 2000)) {
        //delay(2000 - (currenttime - _lastreadtime));
        return true; // return last correct measurement
    }
    
    firstreading = false;
   // Serial.print("Currtime: "); Serial.print(currenttime);
   // Serial.print(" Lasttime: "); Serial.print(_lastreadtime);
    _lastreadtime = millis();
    
    data[0] = data[1] = data[2] = data[3] = data[4] = 0;
    
    // now pull it low for ~20 milliseconds
    pinMode(_pin, OUTPUT);
    digitalWrite(_pin, LOW);
    delay(20);
    cli();
    digitalWrite(_pin, HIGH);
    delayMicroseconds(40);
    pinMode(_pin, INPUT);
    
    // read in timings
    for ( i=0; i< MAXTIMINGS; i++) {
        counter = 0;
        
        while (digitalRead(_pin) == laststate) {
            counter++;
            delayMicroseconds(1);
            
            if (counter == 255)
                break;
        }
        
        laststate = digitalRead(_pin);
    
        if (counter == 255)
            break;
    
        // ignore first 3 transitions
        if ((i >= 4) && (i%2 == 0)) {
            // shove each bit into the storage bytes
            data[j/8] <<= 1;
            
            if (counter > _count)
                data[j/8] |= 1;
                
            j++;
        }
    }
    
    sei();
 
    // check we read 40 bits and that the checksum matches
    if ((j >= 40) &&  (data[4] == ((data[0] + data[1] + data[2] + data[3]) & 0xFF)))
        return true;
    
    return false;
}

    /*==============================================================================
* sendToCloud 
*
* send data to cloud 
*=============================================================================*/
int sendToCloud(String res, int value) 
{
 
  ret = 0;
  stringPos = 0;
  DataRx = false;
  RxLoop = true;
  timeout_time = 0;
  time_now = 0;
  timeout = 3000; // 3 seconds
  myDataString = ""; //allocate for actual data sent

  if (client.connect(serverName,80)) {
    if (client.connected()) {
      myDataString += res;
      myDataString += "="; //put into resource
      myDataString += value; //just send the value
      // Send request using Exosite basic HTTP API
      client.println("POST /api:v1/stack/alias HTTP/1.1");
      client.println("Host: m2.exosite.com");
      client.print("X-Exosite-CIK: "); 
      client.println(cik);
      client.println("Content-Type: application/x-www-form-urlencoded; charset=utf-8");
      client.println("Accept: application/xhtml+xml");
      client.print("Content-Length: ");
      client.println(myDataString.length()); //calculate length
      client.println();
      client.println(myDataString);

      // Read from the nic
      //
      timeout_time = millis()+ timeout;	
      while ((timeout_time > time_now) && RxLoop) { 
        if (client.available()) {
          if (!DataRx)
            DataRx= true;          
          c = client.read();
          rxdata[stringPos] = c;          
          stringPos += 1;
        } else {
          rxdata[stringPos] = 0;

          if (DataRx) {
            DataRx= false;
            RxLoop = false;

            ret=1;
          }
        }//else
        time_now = millis();
      }// while ((timeout_time > time_now) && RxLoop) {

      client.stop();
    }
  }// if (client->connect(serverName,80)) {
  return ret;
}

/*==============================================================================
* readFromCloud 
*
* read data from cloud 
*=============================================================================*/
int readFromCloud(String res ,String* pResult) 
{
 
  ret = 0;
  stringPos = 0;
  DataRx= false;
  RxLoop = true;
  timeout_time = 0;
  time_now = 0;
  timeout = 3000; // 3 seconds
  myDataString = ""; //allocate for actual data sent

  if (client.connect(serverName,80)) {
    if (client.connected()) {
      // Send request using Exosite basic HTTP API
      client.print("GET /api:v1/stack/alias?");
      client.print(res);
      client.println(" HTTP/1.1");
      client.println("Host: m2.exosite.com");
      client.print("X-Exosite-CIK: "); 
      client.println(cik);
      client.println("Accept: application/x-www-form-urlencoded; charset=utf-8");
      client.println();
      // Read from the nic or the IC buffer overflows with no warning and goes out to lunch
      timeout_time = millis()+ timeout;
      
      while ((timeout_time > time_now) && RxLoop) {
        if (client.available()) {
          if (!DataRx)
            DataRx= true;
          
          c = client.read();
          rxdata[stringPos] = c;
          
          stringPos += 1;
        } else {
          rxdata[stringPos] = 0;

          if (DataRx) {
            DataRx = false;
            RxLoop = false;
            String rxstrg = String(rxdata);
            int length = 0;
            int rxresultpos = 0;
            int subStringLength = 0;
            if (rxstrg.startsWith("HTTP/1.1 200 OK")) {
              length = rxstrg.length();
              rxresultpos=rxstrg.indexOf('=');
              subStringLength = length - rxresultpos;
              *pResult= String(rxstrg.substring(rxresultpos+1));
            } else {
              rxresultpos=rxstrg.indexOf('\n');
              subStringLength = rxresultpos;
              *pResult= String(rxstrg.substring(0,rxresultpos));
            }  
            ret=1;
          }
        }
        time_now = millis();
      }
      client.stop();
    }
  }
  return ret;
}

#define DHTPIN D2    // Digital pin D2
#define DHTTYPE DHT22
 
DHT dht(DHTPIN, DHTTYPE);

int h = 0;
int t = 0;

int f = 0;  // failed?


void setup()
{
   Serial.begin(9600);   // open serial over USB
    dht.begin();

}

void loop()
{

 f = 0;

    h = dht.readHumidity() * 10;
    t = dht.readTemperature() * 10;
    
    if (t==NAN || h==NAN)
        f = 1;
    else
        f = 0;

        sendToCloud("H1", h);  
        sendToCloud("T1", t);
        Serial.println(h);
        Serial.println(t);
}

One of my coworkers just sent me a note about you thread.

I’d love to know if you found any workarounds to your issues. An official port of the Exosite library is on my todo list. (I work for Exosite and wrote the current version of the Arduino library.)

I’ve run into problems with the TCPClient that make it impossible to use the existing Arduino library: https://community.spark.io/t/tcpclient-hangs-up-core/946/37 I just tried the example in that thread again this morning and it’s still having problems. As soon as that gets fixed, I’ll be posting the official library.

Hello Azdle!
I solve the problem of delay between two consecutive transmissions using the new library.
But after some time of operation the program freezes, and all the time I have Breathing cyan. After reset the program runs again.
In some cases the core resets itself and continues to work.
It seems the problem is in the TCP library.
In some cases the core resets itself and continues to work.
Also after programming with this program it is difficult to program core via wifi.
Firstly I manually using the dfu to reprogram the firmware. So again can be programmed via WIFI but only once.

New code is the next:
// This #include statement was automatically added by the Spark IDE.
#include “DHT.h”

//*****************************************************************************
//
// exosite.cpp - Prototypes for the Exosite Cloud API
//
// Copyright (c) 2012 Exosite LLC.  All rights reserved.
// 
// Redistribution and use in source and binary forms, with or without 
// modification, are permitted provided that the following conditions are met:

//  * Redistributions of source code must retain the above copyright notice,
//    this list of conditions and the following disclaimer.
//  * Redistributions in binary form must reproduce the above copyright 
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
//  * Neither the name of Exosite LLC nor the names of its contributors may
//    be used to endorse or promote products derived from this software 
//    without specific prior written permission.
// 
// THIS SOFTWARE IS PROVIDED "AS IS" AND WITH ALL FAULTS.
// NO WARRANTIES, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT
// NOT LIMITED TO, IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
// A PARTICULAR PURPOSE APPLY TO THIS SOFTWARE. TI SHALL NOT, UNDER ANY
// CIRCUMSTANCES, BE LIABLE FOR SPECIAL, INCIDENTAL, OR CONSEQUENTIAL
// DAMAGES, FOR ANY REASON WHATSOEVER.
//
//*****************************************************************************
#define serverName        "m2.exosite.com"
#define ACTIVATOR_VERSION  "2.1"
#define ARDUINO  "105"

    TCPClient client;
    char cik[41] = "00000000000000000000000000000000000000"; <-- Fill in your CIK here!
    char rxdata[200];
    char aliasList[50];
    char* varPtr;
    char* varPtr2;
    boolean ret;
    int stringPos;
    boolean DataRx;
    boolean RxLoop;
    char c;
    unsigned long timeout_time;
    unsigned long time_now;
    unsigned long timeout;

// Select a Debug Level: 
//#define EXOSITEDEBUG 1
//#define EXOSITEDEBUG 2
//#define EXOSITEDEBUG 3

/*==============================================================================
* writeRead
*
* One step read and write to Exosite using char arrays.
*=============================================================================*/
boolean writeRead(char* writeString, char* readString, char** returnString){
  ret = false;
  stringPos = 0;
  DataRx= false;
  RxLoop = true;
  timeout_time = 0;
  time_now = 0;
  timeout = 3000; // 3 seconds
  varPtr = aliasList;

  Serial.print("Connecting to Exosite...");

  if (client.connect(serverName,80)) {
    client.flush();
    Serial.println("Connected");

    // Send request using Exosite basic HTTP API
    client.print("POST /onep:v1/stack/alias?");
    client.print(readString);
    client.println(" HTTP/1.1");
    client.println("Host: m2.exosite.com");
    client.print("User-Agent: Exosite-Activator/");
    client.print(ACTIVATOR_VERSION);
    client.print(" Arduino/");
    client.println(ARDUINO);
    client.print("X-Exosite-CIK: "); 
    client.println(cik);
    client.println("Accept: application/x-www-form-urlencoded; charset=utf-8");
    client.println("Content-Type: application/x-www-form-urlencoded; charset=utf-8");
    client.print("Content-Length: ");
    client.println(strlen(writeString)); //calculate length
    client.println();
    client.println(writeString);
    // Read from the nic or the IC buffer overflows with no warning and goes out to lunch
    timeout_time = millis()+ timeout;

    #if EXOSITEDEBUG > 1
      Serial.println("Sent");
    #endif
    
    while ((timeout_time > time_now) && RxLoop) {
      if (client.available()) {
        if (!DataRx)
          DataRx= true;
        
        c = client.read();
        rxdata[stringPos] = c;

        #if EXOSITEDEBUG > 2
          Serial.print(c);
        #endif
        
        stringPos += 1;
      } else {
        #if EXOSITEDEBUG > 1
          Serial.println("No More Data");
        #endif
        rxdata[stringPos] = 0;

        if (DataRx) {
          DataRx = false;
          RxLoop = false;

            #if EXOSITEDEBUG > 1
              Serial.println("HTTP Response:");
              Serial.println(rxdata);
            #endif
  
          if (strstr(rxdata, "HTTP/1.1 200 OK")) {
            #ifdef EXOSITEDEBUG
              Serial.println("HTTP Status: 200");
            #endif
  
            ret = true;
            varPtr = strstr(rxdata, "\r\n\r\n") + 4;

            *returnString = (char*) realloc(*returnString, (rxdata + stringPos + 1) - varPtr);

            if(*returnString == 0)
              break;

            strncpy(*returnString, varPtr, (rxdata + stringPos + 1) - varPtr);
          }else if(strstr(rxdata, "HTTP/1.1 204 No Content")){
            #ifdef EXOSITEDEBUG
              Serial.println("HTTP Status: 204");
            #endif
  
            ret = true;
          } else {
            #ifdef EXOSITEDEBUG
              Serial.println("Warning Unknown Response: ");

              varPtr = strstr(rxdata, "\n");
              *varPtr = '\0';

              Serial.println(rxdata);
            #endif
          }  
        }
      }
      time_now = millis();
    }

    if(timeout_time <= time_now){
      Serial.println("Error: HTTP Response Timeout");
    }
  }else{
    Serial.println("Error: Can't Open Connection to Exosite.");
  }

  client.stop();

  #ifdef EXOSITEDEBUG
    Serial.println("End Char ReadWrite");
  #endif

  return ret;
}

/*==============================================================================
* writeRead
*
* One step read and write to Exosite using Arduino String objects.
*=============================================================================*/
boolean writeRead(String writeString, String readString, String &returnString){
  char *writeCharString, *readCharString, *returnCharString;
  writeCharString = (char*)malloc(sizeof(char) * writeString.length()+1);
  readCharString = (char*)malloc(sizeof(char) * readString.length()+1);
  returnCharString = (char*)malloc(sizeof(char) * 32);

  if(writeCharString == 0 || readCharString == 0 || returnCharString == 0){
    Serial.println("Not Enough Ram! Failing!");
    while(1);
  }

  writeString.toCharArray(writeCharString, writeString.length()+1);
  readString.toCharArray(readCharString, readString.length()+1);

  if(writeRead(writeCharString, readCharString, &returnCharString)){
    returnString = returnCharString;
    ret = true;
  }else{
    Serial.println("Error Communicating with Exosite");
    ret = false;
  }
  free(writeCharString);
  free(readCharString);
  free(returnCharString);

  return ret;
}

/*==============================================================================
* DEPRECIATED METHODS
*=============================================================================*/

/*==============================================================================
* sendToCloud
*
* send data to cloud
*=============================================================================*/
int sendToCloud(String res, int value){
  String readString = "";
  String returnString = "";
  String writeString;
  writeString = res + "=" + value;

  if(writeRead(writeString, readString, returnString)){
    return 1;
  }else{
    Serial.println("Error Communicating with Exosite");
    return 0;
  }
}


/*==============================================================================
* readFromCloud
*
* read data from cloud
*=============================================================================*/
int readFromCloud(String readString ,String* returnString){
  String writeString = "";

  if(writeRead(writeString, readString, *returnString)){
    return 1;
  }else{
    Serial.println("Error Communicating with Exosite");
    return 0;
  }
}

#define DHTPIN D6    // Digital pin D2
#define DHTTYPE DHT22
 
DHT dht(DHTPIN, DHTTYPE);

int h = 0.0;
int t = 0.0;

int f = 0;  // failed?


void setup()
{
Serial.begin(9600);   // open serial over USB
    dht.begin();
    
}


void loop()
{


   f = 0;
  h = dht.readHumidity()*10;
  t = dht.readTemperature()*10;
    
      
    if (t==NAN || h==NAN)
        f = 1;
    else
        f = 0;

        Serial.println(h);
        Serial.println(t);


if((t < NAN) && (h < NAN)) 
{


 String readParam = "T1&H1";
 String writeParam = "";
 String returnString = ""; 
  
   
       writeParam = "T1="; // 
       writeParam += t;
       writeParam +="&H1=";
       writeParam += h;
       writeRead(writeParam, readParam, returnString);
       
}

	/* exosite.sendToCloud("T1", num1);
         exosite.sendToCloud("H1", num2);*/
         //delay(3000);

}

I am sure there are problems with TCPClient, but I have to say that I can run firmware that pulls weather data from Yahoo for many days at a time.

The current Spark memory allocator needs improvement. The Spark team has on their backlog to improve it, but in the mean time, I think you would do better with completely static allocation and never call malloc or realloc or free. I don’t think the core memory allocator recovers gracefully from fragmentation right now.

You can also use the fact that the response data is already buffered in the TCPClient to your advantage and instead of reading it all into another big array for you to handle, you can parse it as you read it. I read the TCP response bytes looking for words separated by spaces or other delimiters. Using a state-machine approach, I never have to store long strings that way.

1 Like

bko, I don’t think that’s my problem. Or at least not my main, direct problem. I created a script that just reads a single URL over and over and has no memory allocations at all, but it still fails. (Presumably, it still has internal memory allocations.) See: https://community.spark.io/t/tcpclient-hangs-up-core/946/38

Out of curiosity, how many calls would you say you get before failing?

Also, I assume you’re using the internal variables in the TCPClient library to read from the existing buffer? I’d rather not code to implementation details, especially because I plan on distributing this code.

Hi Azdle

I saw that other thread but nothing jumped out at me as an area of concern. I also don’t think that malloc is your main problem, but I think removing it will reduce some of the debugging noise you might be getting by failures from different sources.

I can hit http://weather.yahoo.com/forecastrss?w=<<some number>> every 5 minutes for at least several days to a week without a crash, so on the order of thousands GET requests. When it does crash, it is not obvious what is happening and I often do not get a panic code, just a core reset. I light up the D7 LED when I am actually doing the GET request and it does not seem to fail during that part, but after. Once in a while it is just hung and I push the reset button.

What’s different? I pass HTTP/1.0 instead of 1.1. I do not send a User-Agent at all. I do send a Accept: text/html, text/plain. Immediately after I send the last carriage-return, I do flush() to clear out any old input. I would not think any of that matters.

My read uses only the published API looks like this:

    int bytecount = myTCP.available();
    while (bytecount>0) {
        serialEvent();
        bytecount--;
    }

where serialEvent() is my one byte at a time state-machine and does exactly one myTCP.read(). As the bytes come in, I compare them one at a time with a pointer into const char startMatch[] = "<yweather:forecast "; to find the start of the part I want (there are several sections that start with that string and I parse them all). Once I find the starting point, then other parsing takes over like looking for keywords etc. and I do have a buffer and use strtok. I think the buffer is around 64 bytes. But I never read the entire TCP client object buffer into one big buffer in my code, that’s all.

I am not sure what to suggest: maybe you should try your code with a different server to debug? Sometimes weird stuff happens with ec2 servers.

Hello Azdle!
Is there any progress in porting exosite library?

Hey, developer_bt. I haven’t had any time to work on the issue since my last post and I probably won’t have any time in the next month. Not sure about after that. I will definitely make a post here when I have something that I think is mostly usable.

@developer_bt, from the files of “better late than never”, I had one of our interns dig into what the problem was that we were having. Turns out the crashing at boot problem was really simple, there was a Serial.print that was getting called before Serial.begin that just NOPs on the arduino, but causes the spark to crash. We removed that and the library works great on the Photon now (I haven’t actually tested the Core, but I have no reason to think it wouldn’t work). So, it also seems that whatever problem we were having with the stability over time with the TCP library has been fixed.

It’s available in the IDE as “Exosite” otherwise check out the repo here: https://github.com/exosite-garage/particle_exosite_library/

@Azdle really great news :smile:
I already saw that there is a new library for photon / spark but still I do not have tested.
Currently I have only SPARC core, I will test over the weekend.
I hope that the library will work with Electron.

Currently I use the library HttpClient.
The example was written by Azdle,posted on the official Exosite forum.
https://community.exosite.com/t/spark-core-support/72

Thanks.