Need Assistance with creating webhook for Weather Underground

SanDiegoSeth · October 6, 2021, 10:06pm

Hello !!

I am trying to send weather data to Weather Underground from my Particle Boron (which is attached to an IoT Node device), but keep getting an error 400 code each time the webhook is called. According to Weather Underground, the upload protocol should be in the following format:

https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php?ID=YYYYYYYY&PASSWORD=XXXXXX&dateutc=2000-01-01+10%3A32%3A35&winddir=230&windspeedmph=12&windgustmph=12&tempf=70&rainin=0&baromin=29.1&dewptf=68.2&humidity=90

I have created a webhook called "WUBulkWriteCSV" that has 4 form fields, as follows:

ID > {{id}} (for Weather Underground station ID)
PASSWORD > {{pw}} (for Weather Underground password)
dateutc > {{d}} (for current date)
data > {{data}}

===========
Within Particle IDE, I have the following lines of code:

String wuStationID = "KCAVISTA171";
String wuPassword = "XXXXX";
String dateUTC;

String data = "{"ID=":"" + wuStationID + "","PASSWORD=":"" + wuPassword + "","dateutc=":"" + dateUTC + "","data":"" + csvDataWU+ ""}";

Particle.publish("WUBulkWriteCSV", data);

Here is the output of the webhook in "Pretty" form:
{
"ID=": "KCAVISTA171",
"PASSWORD=": "XXXXX",
"dateutc=": "now",
"data": "winddir=270&windspeedmph=0&windgustmph=0&humidity=74&tempf=70.8&rainin=0&dailyrainin=0&baromin=29.48&solarradiation=0"
}

Here is the output of the webhook in "raw" form:
{"ID=":"KCAVISTA171","PASSWORD=":"XXXXX","dateutc=":"now","data":"winddir=270&windspeedmph=0&windgustmph=0&humidity=74&tempf=70.8&rainin=0&dailyrainin=0&baromin=29.48&solarradiation=0"}

============

Can you please help me figure out where the error is coming from? If I compare the upload protocol from Weather Underground with the output of my webhook, I notice that the upload protocol does not include any quotation marks, colons or commas between data fields, but the output of my webhook includes all three of these. In addition, all of their data fields are separated by the "&" symbol, and mine does not have this. I'm not sure how to fix this.

Would really appreciate advice on how to get this to work properly.

Thanks !!

Seth

robc · October 7, 2021, 1:11am

Hi there!! Below I post some code which might help. I have not actually posted data to Weather Underground so I am not certain if I can explain what is wrong. But, I took a stab at demonstrating the use of char arrays instead of String.

Also, here is an interesting and recent post regarding c-strings and char. Thanks to @gusgonnet and all the contributors, btw:

https://community.particle.io/t/best-practices-with-c-strings-char-a-vs-char-b/59639

Now, for the code example. Try your version of the code below in your firmware and see what works for you:

const char myStationId16[16] = "KCAVISTA171";
const char myPassword16[16] = "XXXXXX";
char myDateUtc32[32] = "2000-01-01+10%3A32%3A35"; // over simplified, for sure!
char myWindDirection16[16] = "230";
char myOtherData16[16] = "blah_blah_blah";

//  example of what the data should (might???) look like
char mySendData256[256] = "ID=YYYYYYYY&PASSWORD=XXXXXX&dateutc=2000-01-01+10%3A32%3A35&winddir=230&windspeedmph=12&windgustmph=12&tempf=70&rainin=0&baromin=29.1&dewptf=68.2&humidity=90";

//  display example data buffer
Particle.publish("example_data", mySendData256, PRIVATE, WITH_ACK); 
delay(1000);

// let's build the send data field for real...
sprintf(mySendData256, "ID=%s&PASSWORD=%s&dateutc=%s&winddir=%s&myOtherDataHere=%s", myStationId16, myPassword16, myDateUtc32, myWindDirection16, myOtherData16);   
Particle.publish("test_data", mySendData256, PRIVATE, WITH_ACK); 
delay(1000);

SanDiegoSeth · October 7, 2021, 2:51pm

Hi Rob,

Thanks for your insights. I was able to get the information into the correct format using the strings I already created.

I was able to create the correct string format to match what WU requires by using the following code:

String wuStationID = “KCAVISTA171”;
String wuPassword = “XXXXX”;
String data = ("?ID=" + wuStationID + “&” + “PASSWORD=” + wuPassword + “&” + “dateutc=” + ““now”” + csvDataWU);
Particle.publish(“WUBulkWriteCSV”, data, PRIVATE, WITH_ACK);
delay (1000);

I then modified the webhook called “WUBulkWriteCSV” to only have ONE SINGLE form field, as follows:

data > {{data}}

According to the Events terminal in Particle, the contents sent to the webhook are:

?ID=KCAVISTA171&PASSWORD=XXXXX&dateutc=“now”&winddir=270&windspeedmph=0&windgustmph=0&humidity=77&tempf=69.5&rainin=0&dailyrainin=0&baromin=29.51&solarradiation=0

Although the data being sent to the webhook now matches the format required by Weather Underground, I am still getting the same error, as follows: “error status 400 from weatherstation.wunderground.com”

In addition, there is no longer any output in “pretty” form. All I can see is raw output form.

I think my problem is that the webhook is not recognizing the proper format of my data.

Any guidance you can provide to assist would really be appreciated.

Thanks again,

Seth

robc · October 7, 2021, 3:32pm

I don't have the time to investigate right now, sorry, but maybe others see issues right off the top.

For starters, I do not think the data you formed in the data variable is "proper". For example, this snippet of your data:

&dateutc=“now”

does not show the date in the required utc format. Perhaps this is the reason for your error. In addition, for lack of a better term, the delimiter between your URL and the data or "?" is included in your data. I don't think you should send it twice.

That is all I have for you now, I hope to get back soon.

SanDiegoSeth · October 7, 2021, 5:00pm

Hi Rob,

Thanks for getting back to me. I definitely appreciate your insights.

Please note that I have also generated the string I send to webhooks without the “?” at the beginning of the string and I get the same 400 error. Also, according to Weather Underground, the use of “now” for the dateutc portion of the string is acceptable. So, I think the string should be proper.

When I have generated successful and working webhooks with other projects, I could click on the webhook activity in the Events terminal in Particle and see that when the hook was sent it would generate a “Pretty” form of the webhook. With mine, it does not even generate a “Pretty” version. That is why I think the problem is that the webhook is not properly extracting the data from the string I am sending.

If anybody else has any ideas on how to get this to work, please let me know.

Thanks !!

Seth

gusgonnet · October 7, 2021, 6:01pm

I troubleshoot my webhooks with pipedream
. Sign up for a free account, create a webhook trigger modify the URL of your webhook to point to it momentarily:

best of luck!

robc · October 7, 2021, 6:31pm

Yes, I now see what you mean, according to this link I found on the Weather Underground web site:

https://support.weather.com/s/article/PWS-Upload-Protocol?language=en_US

Instead of:

dateutc=“now”

try this:

dateutc=now

Going back to your original post and using Strings (which I would prefer not to do) here is a change to consider:

// replaced curly quotes with straight quotes
String wuStationID = "KCAVISTA171";
String wuPassword = "XXXXX";
String dateUTC = "now"; //  changed!!  now paired  "dateUTC" : "now"
...
// more changes needed

There is an example, in the link, of a full URL. Perhaps some required fields are missing in your full URL?

dreamER · October 7, 2021, 8:27pm

Hi,
I’m not familiar with webhooks but I make a simple fetch request and looks like there is an issue with Weather Underground API and “CORS” policy as when I call directly the URL I’m getting CORS policy error:

but when I place https://cors-anywhere.herokuapp.com/ on the beginning of URL
and then visit: https://cors-anywhere.herokuapp.com/corsdemo klick on “Request temporary access to the demo server” I’m able to upload the data do Weather Underground with status success:

here is my test script:

<html>
  <head>
 <meta name="viewport" content="width=device-width">
 <meta charset="utf-8"/>
</head>

<style>

 html { 
 background: fixed #8c6aec; 
 -webkit-background-size: cover;
 background-size: cover;
} 
</style>
<body>

<script type="text/javascript">
      var key = "MY_PSWD"; //replace with yours 
      var Id = "MY_ID";  //replace with yours
      
      const url_test = "https://cors-anywhere.herokuapp.com/https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php?ID="+ Id + "&PASSWORD=" + key+ "&dateutc=now&winddir=230&windspeedmph=12&windgustmph=12&tempf=56&rainin=0&baromin=29.1&dewptf=68.2&humidity=90&weather=&clouds=";
// as you can see https://cors-anywhere.herokuapp.com/ is on front of base, Weather Underground URL
async function getData(url) {
  const response = await fetch(url,{
    method: 'GET'
    
  });
 return response.text(); 
   
}
getData(url_test)
.then(data => {
   console.log(data);
   //draw_table(data); 
}).catch(err =>{
   console.log(err);
 });


setInterval(function(){
   getData(url_test);
}, 3000);

</script>
</body>
</html>

Hope that will help some how

SanDiegoSeth · October 9, 2021, 1:03am

Thanks everyone for their insight and assistance thus far.

Good news !! - I no longer have a 400 error

Bad news !! - Now I have a 401 error. Not sure if that is better than a 400 error or not.

Question 1: What are the typical causes of a 401 error?

I really want to use the webhooks function within Particle. This is what I have learned and done so far:

According to Weather Underground, the following three fields are required to send weather data to Weather Underground: ID, PASSWORD, and dateutc. In addition to these three fields, I am also including a 4th field, which is temperature, and a 5th field, which Weather Underground also asks users to include, which is the “action” field.

I have defined five fields in my webhook, as follows:

  "ID": "{{id}}",
  "PASSWORD": "{{p}}",
  "dateutc": "{{t}}",
  "data": "{{tempf}}",
  "action": "{{action}}"

NOTE: Weather Underground upload protocol also states: “IMPORTANT all fields must be url escaped”.

Question #2: If I would like to send the following fields to the webhook, how do I change the following code so that all fields in the webhook are URL escaped and the information is properly recognized by the webhook?

String data = (ID = KCAVISTA171, PASSWORD = XXXX, dateutc=now, data = tempf, action=updateraw);
Particle.publish(“WUBulkWriteCSV”, data, PRIVATE, WITH_ACK);

Any guidance that can be sent my way will be very much appreciated !!

All the best,

Seth

gusgonnet · October 9, 2021, 3:30am

great news! you no longer have an issue with the data in the webhook, but you still have an issue with the credentials, username, password, the fields they contain, or something similar.

HTTP 401

The HyperText Transfer Protocol (HTTP) 401 Unauthorized client error status response code indicates that the client request has not been completed because it lacks valid authentication credentials for the requested resource.
source

can you triple-check that the ID, PASSWORD are correct?
maybe you have a correct ID but it is not the correct ID for the api you are trying to consume?

Cheers

Topic		Replies	Views
[SOLVED] Photon Weather Station not uploading to WUnderground Cloud	23	5695	August 13, 2017
Using WebHooks to Link to Web Services, I thought I had this figured out Troubleshooting	11	2437	September 6, 2017
Webhooks, the console and Ubidots Cloud	31	5934	April 15, 2017
Wunderground webhook Troubleshooting	4	1550	October 23, 2018
Particle.variable() use in a webhook post request? Cloud	22	6014	September 12, 2017

Need Assistance with creating webhook for Weather Underground

Particle.publish("WUBulkWriteCSV", data);

Related topics