Webhooks and jumping through hoops :/

Troubleshooting
1

I have been working on a solar powered Particle Electron project and a Vapor 3.0 (Swift) based back-end. While the Electron itself is a great platform, getting the data in the back-end feels quite cumbersome.

When you want to publish a single value, all is fine (well, ‘fine’ because this still has to a cast to String for Particle.publish to work properly :man_facepalming:t2:). But when you want to do anything a little more complicated, things start breaking and leading to non-compliant requests.

One of the things I added was keeping track of battery / charge status, trying to send a batch of data to the backend. Because it’s not possible to send multiple values in one Publish event (other than concatenating them into a string) I went the nested JSON route:

/// Battery and Solar Charging Status

float batteryChargePercentage = 0.0;    // initial value

void readPowerStatus() {
    FuelGauge fuel;
    PMIC power_management;
    
    // get the battery percentage
    float chargePercentage = fuel.getSoC();
    
    // only submit every 2% change
    if (abs(chargePercentage - batteryChargePercentage) >= 2) {
        Serial.printlnf("Battery: %.4f v (%.4f %% charged, has power: %d)", fuel.getVCell(), fuel.getSoC(), power_management.isPowerGood());
        
        // battery voltage, state of charging and whether or not we have usb power
        // {"v": "4.05", "soc": "77.81", "c": true, "h": false}
        char publishString[64];
        sprintf(publishString, 
            "{\"v\": %.4f, \"soc\": %.4f, \"c\": %s, \"h\": %s}",   // the JSON to publish
            fuel.getVCell(),                                        // battery voltage, e.g. 3.86
            chargePercentage,                                       // battery State of Charge, e.g. 44.87
            (power_management.isPowerGood() ? "true" : "false"),    // whether or not we have USB power from solar panel (0/1)
            (power_management.isHot() ? "true": "false"));          // whether or not In Thermal Regulation (bad!)
        Particle.publish("b", publishString, PRIVATE);
        
        // remember batter percentage
        batteryChargePercentage = chargePercentage;
    }
}

I created a web hook that would trigger on this event and call my RESTful Vapor based back-end:

url: http://myubuntuserver/cores/{{PARTICLE_DEVICE_ID}}/charges
payload:

{
  "battery": "{{PARTICLE_EVENT_VALUE}}",
  "published_at": "{{PARTICLE_PUBLISHED_AT}}"
}

The issue here is that the moustache templates enforce that you need to send everything as a string, which basically means that the nested JSON I am trying to send becomes a String.

So instead of the expected (or wanted) payload:

{
	"battery": {
		"v": 4.05, 
		"soc": 77.81, 
		"c": true, 
		"h": false
	},
	"published_at": "2018-02-22T11:39:15.159Z"
}

This becomes:

{
	"battery": "{\"v\": 3.9950, \"soc\": 76.8320, \"c\": false, \"h\": false}",
	"published_at": "2018-02-22T11:39:15.159Z"
}

Which basically results in having to double parse incoming requests :persevere:

This is already a workaround, because I would rather have the following payload without having to go nested JSON altogether:

{
	"v": 4.05, 
	"soc": 77.81, 
	"c": true, 
	"h": false,
	"published_at": "2018-02-22T11:39:15.159Z"
}

And this is not possible either because Publish only takes Strings. So this would become

{
	"v": "4.05", 
	"soc": "77.81", 
	"c": "true", 
	"h": "false",
	"published_at": "2018-02-22T11:39:15.159Z"
}

Where you would have -again- to add in additional transformations for changing Double-As-A-String into Doubles and Boolean strings into booleans… :cold_sweat:

Sorry, but I am getting pretty frustrated with having to jump through so many hoops to get something -in my eyes- pretty simple working… :exploding_head: Are moustache templates really the right solution here?

2

You’ll need to use the body feature of the webhook instead of the json feature. Then you’d set to body to:

"body":"{ \"v\":{{v}}, \"soc\":{{soc}}, \"c\":{{c}}, \"h\":{{h}}, \"published_at\": \"{{PARTICLE_PUBLISHED_AT}}\" } "

Continue to pass the data in the same way in the Particle.publish, but now your server will get a true, correct JSON object with real numbers and booleans in it:

{
    "v": 4.05,
    "soc": 77.81,
    "c": true,
    "h": false,
    "published_at": "2018-03-02T16:15:26.671Z"
}

These tips might help:

5 Likes
3

Oh man, thanks for your quick reply @rickkas7, I was getting so frustrated :slight_smile: I have things working! :raising_hand_man:t2:

To document how I got it working:

Electron firmware:
Unchanged from the code in the start post.

Web Hook Full URL
Is still the same: http://my.server/cores/{{PARTICLE_DEVICE_ID}}/charges

Custom Request Body:

{"battery": {"v":{{v}},"soc":{{soc}},"c":{{c}},"h":{{h}}},"published_at":"{{PARTICLE_PUBLISHED_AT}}"}

HTTP Header:
In order to be able to properly decode the custom JSON body, setting the header is important:
Content-Type: application/json, although this probably depends on your back-end and the framework you are using.

This request lands on the server as:

{"battery": {"v": 4.05,"soc": 77.81,"c": true, "h": false},"published_at": "2018-03-02T17:03:40.125Z"}

And Swift's JSON Decoder is able to decode the body to Codable structs :slight_smile:

I might now get rid of the nesting altogether, but at least this works :bowing_man:t2:

This really needs some proper documentation in the particle docs though…

Below is the Model for this JSON which I use in Vapor 3.0 (currently in Beta):

import Foundation
import FluentPostgreSQL
import Vapor

// Example JSON
// ------------
// {
//    "battery": {
//        "v": 4.05,
//        "soc": 77.81,
//        "c": true,
//        "h": false
//    },
//    "published_at": "2018-02-22T11:39:15.159Z"
// }

struct BatteryData: PostgreSQLJSONType, Codable {
    var voltage: Double                 // battery voltage
    var stateOfCharging: Double         // percentage charged
    var isCharging: Bool = false        // whether or not VUSB is providing power
    var isInThermalRegulation = false   // whether or not we're in thermal regulation
    
    enum CodingKeys: String, CodingKey {
        case voltage = "v"
        case stateOfCharging = "soc"
        case isCharging = "c"
        case isInThermalRegulation = "h"
    }
}

final class BatteryCharge: Codable {
    var id: Int?
    var coreID: Core.ID?
    var battery: BatteryData
    var publishedAt: Date?

    /// Initializer
    init(id: Int? = nil, coreID: Core.ID?, battery: BatteryData, publishedAt: Date) {
        self.id = id
        self.coreID = coreID
        self.battery = battery
        self.publishedAt = publishedAt
    }
    
    enum CodingKeys: String, CodingKey {
        case id
        case coreID
        case battery
        case publishedAt = "published_at"
    }
}

/// MARK: Model
extension BatteryCharge: Model {
    /// See Model.Database
    typealias Database = PostgreSQLDatabase

    /// See Model.ID
    typealias ID = Int

    /// See Model.idKey
    static var idKey: WritableKeyPath<BatteryCharge, Int?> {
        return \BatteryCharge.id
    }
}

/// MARK: Relations
extension BatteryCharge {
    /// A relation to this battery's core
    var core: Parent<BatteryCharge, Core>? {
        return parent(\.coreID)
    }
}

/// Support dynamic migrations.
extension BatteryCharge: Migration { }

/// Support encoded to and decoded from HTTP messages.
extension BatteryCharge: Content { }

/// Support using as a dynamic parameter in route definitions.
extension BatteryCharge: Parameter { }
4 Likes