I picked up a project that I was working on a while back. Since I left it, the NOAA rss feed that my Photon requests data from changed from HTTP to HTTPS. When I try to connect to the feed using the TCP client, my terminal results print “HTTP/1.1 301 moved permanently (among other things).”
Is there a fix for this with the TCP client? Or do I need to look at capturing the data in some other way. Thanks for reading!
I believe you have to transition to using a webhook when requesting data from HTTPS URLs. If you search for "HTTPS " you'll find some older threads asking about making HTTPS requests and that it is not possible on the Photon hardware. Hence the need for the webhook. This thread is kind of old but I believe it is still true:
Although, it looks like there is a library available if you want to still keep the process local to the Particle device:
Thanks for the quick response @ninjatill! I’ll look into webhooks. Do you know if they can return the data in the same format as the TCP client? I have a lot of code that parses the NOAA data to use for servos and leds and it would be great if I can reuse that. Thanks again.
Sorry, I don’t know enough about webhooks to guide you further. Although, having a sample of the data or a link to a sample would help me, or someone else, give you some suggestions. Perhaps @rickkas7 or @ScruffR could help.
Webhooks return their responses as a Cloud event which can be subscribed to via Particle.subscribe(). With any Device OS before 0.8.0 the max payload for a single response block is 255 “characters” (not all bytes are allowed) and with 0.8.0+ 622. But longer responses can be caught as multi-part.
As @ninjatill said, having an idea what your respone would look like would make it easier to tell whether it’s going to work or not.
Thanks guys. I am attempting to grab data about ocean conditions here - https://www.ndbc.noaa.gov/data/latest_obs/46232.rss
Will that be a good fit for webhooks? Previously, I would store the data I was interested in using a char array (I think… Its been a while since I was working on this and may be mixing my terminology.) I was not using String.
If you went for https://www.ndbc.noaa.gov/data/latest_obs/46232.txt the response would be much more concise and definetly fit a single webhook response.
That looks way easier to handle. I didn’t know those txt formats existed. Thank you @ninjatill @ScruffR You saved me a lot of time by pointing me in the right direction!
I wanted to play around so I created a webhook using the original URL you posted. I simply output the response to the serial monitor every 60 seconds. If you already have a parsing routine for the existing format, then you should have no problem taking the webhook reponse and feeding it into your parser.
To setup the webhook, open the console and click Integrations --> New Integration --> Webhook and set it up like this:
Here is the code and output:
void setup() {
// Subscribe to the integration response event
Particle.subscribe("hook-response/NOAA_Station46232_PointLomaSouthCA", myHandler, MY_DEVICES);
Serial.begin(9600);
delay(10000);
}
void myHandler(const char *event, const char *data) {
// Handle the integration response
Serial.print(data);
}
void loop() {
// Get some data
const char *data;
// Trigger the integration
Particle.publish("NOAA_Station46232_PointLomaSouthCA", data, PRIVATE);
// Wait 60 seconds
delay(60000);
}
As @ScruffR said, if you switch to the .txt file, it may be more concise to pull the data from that file than deal with the XML/HTML ouput.
This is awesome. Thank you for showing the integration, code, and output!
Hi @ninjatill and @ScruffR. I’m having trouble with something that seems like it should be straight forward. My old code used a function to compare a char returned by the TCP client to know if the values should be ignored or used… but when I’m working with the webhook it returns the data as a const char.
I’m not sure if I should be trying to convert the returned ‘const char’ or if I need to copy it as another data type and then use it… Can you let me know the way I should be approaching this?
void stringHandler() {
char c = client.read();
Serial.print(c);
if (c == SS) {
tagStarted = true;
tagEnded = false;
tagIndex = 0;
tagData[tagIndex] = '\0';
}
// Is it the end of a strong tag?
else if (c == SE) {
tagEnded = true;
if (strcmp(tagData, STRONGSTART) == 0) {
strongStarted = true;
strongEnded = false;
//Serial.println("STRONG tag found...");
}
else if (strcmp(tagData, STRONGEND) == 0) {
strongEnded = true;
//Serial.println("End of STRONG tag found...");
parseStrong();
strongIndex = 0;
strongData[strongIndex] = '\0';
}
}
// Is it part of a tag?
else if (tagStarted && !tagEnded) {
if (tagIndex < tagLen)
{
tagData[tagIndex++] = c;
tagData[tagIndex] = '\0';
}
}
// Is it part of a strong tag?
else if (strongStarted && !strongEnded) {
if (strongIndex < strongLen)
{
strongData[strongIndex++] = c;
strongData[strongIndex] = '\0';
}
}
} //end stringHandler
const char* just means the value cannot and mustnot be altered, but a mere string compare would not do that anyway, so no problem there.
But for that ...
... you wouldn't need to compare the whole string but only representative portions and that would be easier done with strstr() and strcmp().
When you use the plain text output, parsing should be rather simple.
You can also use sscanf() for parsing the text - only be aware that %f is not (currently) supported on Particle devices. You need to parse these as "string" with the something like %[-0123456789.] and a subsequent conversion via atof()