Webhooks and Ubidots

I have been using Ubidots and Webhooks for some time on the Electron but, I wanted to try using the “Form” approach and have hit a brick wall.

I will post the code at the end but, here is the bottom line.

  1. I have created a WebHook called soilMoisture:

  2. I have included the simplest possible code in my application:
    String data = String(90);
    Particle.publish(“soilMoisture”, data ,PRIVATE);
    NonBlockingDelay(3000);

  3. I can see this data going out to Ubidots in the console:

and I know Ubidots is getting something as it created a new device.

In fact, if I send this CURL command:

curl -X POST -H "Content-Type: appl.com/api/v1.6/devices/(my Particle Device ID)/?token=(my Token) -d ‘{“moisture”:400}’

I get the 201 code back from Ubidots and a new variable and datapoint are created. This Form based WebHook integration looks so slick in the tutorials. Why can’t I make it work?

Any ideas?

Thanks,

Chip

Here is my complete code: @scruffr I have made progress in eliminating Strings but not quite there.

/*
 * Project AquaMaster
 * Description: Watering program for the back deck
 * Author: Chip McClelland
 * Date: 7/18/17
 */

 STARTUP(WiFi.selectAntenna(ANT_AUTO)); // continually switches at high speed between antennas
 SYSTEM_THREAD(ENABLED);

 // Finally, here are the variables I want to change often and pull them all together here
 #define SOFTWARERELEASENUMBER "0.25"

 #include <I2CSoilMoistureSensor.h>

 I2CSoilMoistureSensor sensor;

 const int solenoidPin = D2;
 const int blueLED = D7;
 const int donePin = D6;              // Pin the Electron uses to "pet" the watchdog
 const int wakeUpPin = A7;            // This is the Particle Electron WKP pin


 unsigned long waterTimeShort = 60000;    // For Testing the system and smaller adjustments
 unsigned long waterTimeLong = 300000;    // The longest watering period
 unsigned long waterTime = 0;             // How long with the watering go in the main loop
 int startWaterHour = 6;                  // When can we start watering
 int stopWaterHour = 8;                   // When do we stop for the day
 char RSSIdescription[17];
 char RSSIstring[5];
 char capDescription[12];
 int capValue = 0;      // This is where we store the soil moisture sensor raw data
 int wateringNow = 0;
 int waterEnabled = 1;
 int currentPeriod = 0;  // Change length of period for testing 2 times in main loop
 int lastWateredPeriod = 0; // So we only wanter once an hour
 const char* releaseNumber = SOFTWARERELEASENUMBER;  // Displays the release on the menu
 volatile bool doneEnabled = true;    // This enables petting the watchdog
 volatile bool watchdogPet = false;

void setup() {
  pinMode(donePin,OUTPUT);       // Allows us to pet the watchdog
  digitalWrite(donePin, HIGH);  // Pet now while we are getting set up
  digitalWrite(donePin, LOW);

  Particle.variable("Watering", wateringNow);
  Particle.variable("WiFiStrength", RSSIdescription);
  Particle.variable("RSSI",RSSIstring);
  Particle.variable("Moisture", capDescription);
  Particle.variable("capValue", capValue);
  Particle.variable("Enabled", waterEnabled);
  Particle.variable("Release",releaseNumber);
  Particle.function("start-stop", startStop);
  Particle.function("Enabled", wateringEnabled);
  Particle.subscribe("hook-response/soilMoisture", myHandler, MY_DEVICES);      // Subscribe to the integration response event
  //Particle.subscribe("hook-response/watering", myHandler, MY_DEVICES);      // Subscribe to the integration response event


  Wire.begin();
  Serial.begin(9600);
  sensor.begin(); // reset sensor
  NonBlockingDelay(2000);

  pinMode(donePin,OUTPUT);       // Allows us to pet the watchdog
  attachInterrupt(wakeUpPin, watchdogISR, RISING);   // The watchdog timer will signal us and we have to respond
  pinMode(solenoidPin,OUTPUT);
  digitalWrite(solenoidPin, LOW);
  pinMode(blueLED,OUTPUT);
  pinMode(wakeUpPin,INPUT_PULLDOWN);   // This pin is active HIGH

  Particle.connect();    // <-- now connect to the cloud, which ties up the system thread


  Serial.println("");                 // Header information
  Serial.print(F("AquaMaster - release "));
  Serial.println(releaseNumber);

  pinMode(solenoidPin,OUTPUT);
  digitalWrite(solenoidPin, LOW);
  pinMode(blueLED,OUTPUT);

  Time.zone(-4);    // Raleigh DST (watering is for the summer)

  Serial.print("I2C Soil Moisture Sensor Address: ");
  Serial.println(sensor.getAddress(),HEX);
  Serial.print("Sensor Firmware version: ");
  Serial.println(sensor.getVersion(),HEX);
  Serial.println();
}


void loop() {
  if (Time.hour() != currentPeriod)
  {
    getMoisture();
    Particle.publish("Soil Temp C",String(sensor.getTemperature()/(float)10),3); //temperature register
    getWiFiStrength();
    currentPeriod = Time.hour();
    if (waterEnabled)
    {
      if (currentPeriod >= startWaterHour && currentPeriod <= stopWaterHour)
      {
        if ((strcmp(capDescription,"Very Dry") == 0) || (strcmp(capDescription,"Dry") == 0) || (strcmp(capDescription,"Normal") == 0))
        {
          if (currentPeriod != lastWateredPeriod)
          {
            lastWateredPeriod = currentPeriod; // Only want to water once an hour
            if (currentPeriod == startWaterHour) waterTime = waterTimeLong;
            else waterTime = waterTimeShort;
            turnOnWater(waterTime);
          }
        }
        else Particle.publish("Watering","Not Needed");
      }
      else Particle.publish("Watering","Not Time");
    }
    else Particle.publish("Watering","Not Enabled");
  }
}

void turnOnWater(unsigned long duration)
{
  Particle.publish("Watering","Watering");
  digitalWrite(blueLED, HIGH);
  digitalWrite(solenoidPin, HIGH);
  wateringNow = 1;
  NonBlockingDelay(duration);
  digitalWrite(blueLED, LOW);
  digitalWrite(solenoidPin, LOW);
  wateringNow = 0;
  Particle.publish("Watering","Done");
}

int startStop(String command)   // Will reset the local counts
{
  if (command == "start")
  {
    turnOnWater(waterTimeShort);
    return 1;
  }
  else if (command == "stop")
  {
    digitalWrite(blueLED, LOW);
    digitalWrite(solenoidPin, LOW);
    wateringNow = 0;
    Particle.publish("Watering","Done");
    return 1;
  }
  else
  {
    Serial.print("Got here but did not work: ");
    Serial.println(command);
    return 0;
  }
}

int wateringEnabled(String command)
  {
    if (command == "enabled")
    {
      waterEnabled = 1;
      return 1;
    }
    else if (command == "not")
    {
      waterEnabled = 0;
      return 1;
    }
    else
    {
      waterEnabled = 0;
      return 0;
    }
  }

int getWiFiStrength()
{
  int wifiRSSI = WiFi.RSSI();
  strcpy(RSSIstring,String(wifiRSSI));
  if (wifiRSSI >= 0)
  {
    strcpy(RSSIdescription,"Error");
    return 0;
  }
  int strength = map(wifiRSSI, -127, -1, 0, 5);
  switch (strength)
  {
    case 0:
      strcpy(RSSIdescription,"Poor");
      break;
    case 1:
      strcpy(RSSIdescription, "Low");
      break;
    case 2:
      strcpy(RSSIdescription,"Medium");
      break;
    case 3:
      strcpy(RSSIdescription,"Good");
      break;
    case 4:
      strcpy(RSSIdescription,"Very Good");
      break;
    case 5:
      strcpy(RSSIdescription,"Great");
      break;
  }
}


void getMoisture()
{
  capValue = sensor.getCapacitance();
  String data = String(90);
  Particle.publish("soilMoisture", data ,PRIVATE);
  NonBlockingDelay(3000);
  if ((capValue >= 650) || (capValue <=300))
  {
    strcpy(capDescription,"Error: ");
  }
  else
  {
    int strength = map(capValue, 300, 650, 0, 5);
    switch (strength)
    {
      case 0:
        strcpy(capDescription,"Very Dry: ");
        break;
      case 1:
        strcpy(capDescription,"Dry: ");
        break;
      case 2:
        strcpy(capDescription,"Normal: ");
        break;
      case 3:
        strcpy(capDescription,"Wet: ");
        break;
      case 4:
        strcpy(capDescription,"Very Wet: ");
        break;
      case 5:
        strcpy(capDescription,"Waterlogged: ");
        break;
    }
  }
  strcat(capDescription,String(capValue));
  Particle.publish("Moisture Level", capDescription);
}

void NonBlockingDelay(int millisDelay)  // Used for a non-blocking delay
{
    unsigned long commandTime = millis();
    while (millis() <= millisDelay + commandTime) { }
    return;
}

void myHandler(const char *event, const char *data)
{
  if (!data) {              // First check to see if there is any data
    Particle.publish("WebHook", "No Data");
    return;
  }
  String response = data;   // If there is data - copy it into a String variable
  int datainResponse = response.indexOf("moisture") + 24; // Find the "hourly" field and add 24 to get to the value
  String responseCodeString = response.substring(datainResponse,datainResponse+3);  // Trim all but the value
  int responseCode = responseCodeString.toInt();  // Put this into an int for comparisons
  switch (responseCode) {   // From the Ubidots API refernce https://ubidots.com/docs/api/#response-codes
    case 200:
      Serial.println("Request successfully completed");
      doneEnabled = true;   // Successful response - can pet the dog again
      digitalWrite(donePin, HIGH);  // If an interrupt came in while petting disabled, we missed it so...
      digitalWrite(donePin, LOW);   // will pet the fdog just to be safe
      break;
    case 201:
      Serial.println("Successful request - new data point created");
      doneEnabled = true;   // Successful response - can pet the dog again
      digitalWrite(donePin, HIGH);  // If an interrupt came in while petting disabled, we missed it so...
      digitalWrite(donePin, LOW);   // will pet the fdog just to be safe
      break;
    case 400:
      Serial.println("Bad request - check JSON body");
      break;
    case 403:
      Serial.println("Forbidden token not valid");
      break;
    case 404:
      Serial.println("Not found - verify variable and device ID");
      break;
    case 405:
      Serial.println("Method not allowed for API endpoint chosen");
      break;
    case 501:
      Serial.println("Internal error");
      break;
    default:
      Serial.print("Ubidots Response Code: ");    // Non-listed code - generic response
      Serial.println(responseCode);
      break;
  }
}

void watchdogISR()
{
  if (doneEnabled)
  {
    digitalWrite(donePin, HIGH);
    digitalWrite(donePin, LOW);
    watchdogPet = true;
  }
}

@chipmc, looking at your code I have a few “fundamentals” comments:

  1. Your NonBlockingDelay() function is actually blocking! During the while(), I recommend calling Particle.process() to allow user-thread background processes to run. This function is most likely a large contributor to your issues.

  2. You are running in AUTOMATIC mode but also SYSTEM_THREAD(ENABLED) so calling Particle.connect() won’t do anything. If you want to control WiFi and Cloud connectivity, run in MANUAL or SEMI-AUTOMATIC modes.

Make sure that your code is not calling Particle.publish() more than once per second or, at most, in burst of no more than 4 per second with a 4 second pause in between bursts.

1 Like

Hello @chipmc,

The Ubidots guide provide to the users how to setup the “Form” approach for the Webhook. I made a sample test following the steps described above, and I’m not able to reproduce you issue. I have implemented the same configurations, and using the sample code provided:

  void loop() {
  // Get some data
  String data = String(10);
  // Trigger the integration
  Particle.publish("soilMoisture", data, PRIVATE);
  // Wait 60 seconds
  delay(60000);
}

As you can see I’m getting the right response code and the data is received in the Ubidots side:

Please, to discard where can be the issue, try using the sample code provided to verify if works properly. If so, please add in small fragments your routine to the base example in order to verify where can be the error.

All the best,
Maria C.

@peekay123,

Thank you for taking a look at my code. I have made the changes you suggested:

  1. Made my NonBlockingDelay nonblocking:
void NonBlockingDelay(int millisDelay)  // Used for a non-blocking delay
{
    unsigned long commandTime = millis();
    int processFrequency = 1000;
    unsigned long lastProcess = 0;
    while (millis() <= millisDelay + commandTime) {
      if (millis() >= lastProcess + processFrequency)
      {
        Particle.process();
        lastProcess = millis();
      }
    }
    return;
}
  1. Removed the Particle.connect();

  2. Reduced the number of Particle.publish events so I will stay under the 4 per second rule.

No change to the issue with Ubidots but I see the merit of these. Thank you!

Chip

@chipmc, it is safe to call Particle.process() continuously without the 1 second delay you implemented:

while (millis() <= millisDelay + commandTime) Particle.process;

Maria,

Thanks for taking a look.

Two questions:

  1. How important is the 60 second delay? What if my program simply continues on to other things.

  2. Can anything be learned by looking at the data in the Console Log such as this response from Ubidots:

{“data”:"{“moisture”: [{“status_code”: 400, “errors”: {“value”: [“A valid number is required.”]}}]}",“ttl”:60,“published_at”:“2017-07-28T14:47:35.749Z”,“coreid”:“particle-internal”,“name”:“hook-response/soilMoisture/0”}

Thanks,

Chip

@peekay123,

Even better, thanks.

Chip

Don’t worry about it, we’re here to help you! :slight_smile:

Answering your questions:

  1. As I said before, for testing purpose I used the sample code provided by Particle once you created the Webhook, but you can set the delay required for your application and you shouldn’t get any issue.

  1. The status code response 400 means Bad Request – Error due to an invalid body in your request. Please verify if it’s a valid JSON string and that the fields are the ones expected by the endpoint (string, object or float).

See the REST API Reference for any doubt with it!

Best Regards,
Maria C.

@mariahernandez,

OK, sounds like a good step. How can I see the JSON string that is being sent? I don’t see it in the Particle Console.

Also, can you verify that the screen shots I attached to this post look correct?

Thanks,

Chip

@chipmc,

Unfortunately I’m not sure about that… Because the particle packages goes encrypted. For that reason I recommend you start with the sample code which works properly, and then you can add in small fragments your routine to the base example in order to verify where can be the error.

Maria C.

Send you hook to https://requestb.in/ or simailr and that way you can see exactly what you are sending

1 Like

@Viscacha

Thank you for the suggestion.

OK, I gave up on the “Table” integration after seeing others had done the same. I am using the custom JSON approach and was able to send multiple data points in one publish. Great news!

However, I wanted to add “context” to one of the values and I am stuck.

After reading the Ubidots API reference, I think the format I need to add “context” to the Watering variable looks like this in my Webhook:

{
  "Moisture": "{{Moisture}}",
  "Watering": {
    "value": "{{Watering}}",
    "context": "{{status}}"
  },
  "SoilTemp": "{{SoilTemp}}"
}
  

I put together a string that I think fits the bill with this line of code:

String status = "Oops";
String data = String::format("{\"Moisture\":%i, \"Watering\": { \"value\": %i, \"context\":\"" + status + " \"}, \"SoilTemp\":%i}",capValue, wateringMinutes, soilTemp);
Particle.publish("AquaMaster", data, PRIVATE);

In the console, I see the web hook go out and it looks OK:

{"data":"{\"Moisture\":517, \"Watering\": { \"value\": 0, \"context\":\"Oops \"}, \"SoilTemp\":16}

but when I look at what is actually sent to Ubidots using Requestb.in - I see the raw data looks like this:

{"SoilTemp":"16","Watering":{"context":"","value":"[object Object]"},"Moisture":"517"}

What is happening here? How did the JSON get changed from what is listed in the console to what was sent to Ubidots?

Thanks,

Chip

I’m literally about to try much the same thing myself so don’t have a quick answer for you @rikkas7 might be the man to ask as he wrote this pretty extensive article on JSON https://github.com/rickkas7/particle-webhooks . JSON, webhooks etc are all new to me too so I’ve been stumbling around like you. If I was to guess the “Interpreter” on Particle that turns your published string into some JSON before sending it to Ubidots needs some quotes or escape characters in there somewhere to properly convert your string. I’m pretty sure I saw an example that used context however, possibly in here?

@Viscacha,

Thank you for the encouragement. I would like to figure this out. I have been going through @rickkas7 tutorial and the firmware docs and it may be that adding context is more complicated than what the Particle Webhook - JSON approach can handle. I may need to use the “body” approach and this is covered in Rick K’s tutorial.

@mariahernandez or @aguspg I need your help on the format of the request as I can’t seem to find a good example of context for a variable outside of Lat and Long. Using, one of those as a starting point, I have constructed the following CURL request (edited for privacy):

curl -XPOST -H 'Content-Type: application/json;' -H 'X-Auth-Token: (my token)' -d '{"SoilTemp":"19","Watering":{"value": "0", "context": "Oops"},"Moisture":"516"}' https://things.ubidots.com/api/v1.6/devices/(my device API Label)

I am getting a weird response that I need to figure out. I have searched the Ubidots API reference and forums and am coming up empty. Here is the response:

{"watering": [{"status_code": 400, "errors": {"context": ["'Oops' value must be a dict."]}}], "soiltemp": [{"status_code": 204}], "moisture": [{"status_code": 204}]}

I could not find a reference to error code 204 in the response code documentation and I could find no reference to “”‘Oops’ value must be a dict."". Can you please tell me what I am doing wrong.

My hope is that I can solve this from both ends: get the curl command right and then figure out how to create a “body” style Webhook from the command line interface.

Any help or suggestions are appreciated.

Chip

HTTP response codes are standardised, so no need for redundant docs by Particle or Ubidots IMO (when they do, it’ll be to add extra meaning for their special cases)
https://httpstatuses.com/

204 = (“success” as all 2xx codes - but …) no content

@ScruffR,

Thank you for the link, I have bookmarked it for future reference.

I need to better understand how to troubleshoot these issues. For example, status code 204 means success but “no content” for SoilTemp yet, I sent what looks like a valid key / value pair: “SoilTemp”:“16”

I appreciate your help.

Thanks,

Chip

I’m not sure, but how about case sensitivity (SoilTemp vs. soiltemp & Moisture vs. moisture)?

@ScruffR,

I looked at that and it seems that Ubidots takes SoilTemp and makes it lower case as the Webhook send with Caps but Ubidots assigns it an all lower case API label. This is from the JSON Webhook I am using now and that works perfectly.

Still, it is likely a little thing like a quote or a delimiter that is causing this to fail so I appreciate your input.

Chip

Greetings @chipmc, looping into your issue you are building in a wrong way your context, the context is a key-value dictionary, so your payload should be something like this:

{"SoilTemp":"19","Watering":{"value": "0", "context": {"key1":"Oops"}},"Moisture":"516"}' https://things.ubidots.com/api/v1.6/devices/{DEVICE_LABEL}

So please try with this command:

curl -X POST -H 'Content-Type: application/json;' -H 'X-Auth-Token: {PUT_HERE_YOUR_TOKEN} -d '{"SoilTemp":"19","Watering":{"value": "0", "context": {"key1":"Oops"}},"Moisture":"516"}' https://things.ubidots.com/api/v1.6/devices/{PUT_HERE_YOUR_DEVICE_LABEL}

Regards

1 Like

@jotathebest,

Thank you! That worked, got the response code:
{“watering”: [{“status_code”: 201}], “soiltemp”: [{“status_code”: 201}], “moisture”: [{“status_code”: 201}]}

But, when I look at the data point in Ubidots, the “Context” column is blank. How do I display this context value in the Ubidots UI?

Chip