I have two Electron-based devices that have been in the field for months. They normally use 5-10 MB per month of bandwidth. However, both were paused today for supposedly using ~85 MB of bandwidth in the current month (since June 20th). Any thoughts on why things would change so radically? They are both running 1.5.1 and the same application code that has not been problematic in the past.
There seems to be another user seeing a similar thing
I have the same issue here
Me too. The device blew through the 5Mb limit until I caught it at over 35Mb. I wish the data limit mechanisms at Particle had faster reaction times.
Did the data limit kick in at all?
I remember reporting the data limits I set on the Boron not working at all along time ago.
could all this be related to the huge outage of today?
1 Like
Data limits did not work. They were set at 10MB. Regardless, there has to be another issue, as these devices only send data once per minute and there’s no way they could have sent this much data. Yes, @gusgonnet, I believe it’s due to yesterday’s issue, but I don’t want to “assume” that. 
Data limits it did work with me and i set them up to 100MB but I am worried about the cost now (next bill)
@majj_11, I don’t know if I’d do that. Yes, it will likely bring your device back online, but you will have established that as an acceptable limit and I’d think you’ll be responsible for the excessive data usage. It will be interesting to see how Particle handles this issue.
1 Like
@RWB,
In my case it did not even though the 5Mb limit was exceeded by 30Mb over 24 hours. I think the particle data use limits only trip when you exceed them sloooowly.
Chip
I too saw this problem on a few of our devices. They are consistent higher data users, but I’m not sure the amounts allegedly used are accurate. These devices have been in the field > 2 years, and have never used this amount of data to my knowledge.
Another one … this is going to get expensive.
Can we ask for usage credits since the limit only kicked in after the device went over by 23Mb? Also, any idea why this has started happening?
I submitted a support ticket yesterday and got a reply that they will be giving credits related to this issues since it’s related to their problems Wed-Thurs this week. I had to bump my limit to 100MB in order to get my (2) devices back online and the thought of getting hit with that bill was a bit frightful.
I did a little digging and found a clue as to why this is happening - at least to my devices.
Some of my devices have stopped getting Webhook responses back after the outage.
I use a response template in my Webhooks to take the response from Ubidots down to just the HTTP response code (ie 201). For some devices, this response is not coming through even though the Webhook goes out and Ubidots creates the data point. When there is not a response within 45 seconds, the device goes into an error state and reboots only to restart the process again and again - which chews up data more quickly than the data cap functionality can react. Could some devices have corrupted entries that are screwing up the Response Topic (which includes the Particle device ID and the Webhook name.
I also found a simple fix. I flashed Tinker onto the devices and then refreshed them with the original firmware. This cleared up the issue immediately.
Hope this helps,
Chip
3 Likes
Interesting.
I have given up on a simple watchdog concept, to provoke a handshake with the cloud from the device, in order to shorten these outages.
Inspired by posts about skipping the cloud all together, instead I am considering a fall back approach skipping the particle cloud when in trouble.
Normally the particle cloud is used as a go between with a confirm from the receiving server at the other end back to the device. Saves data compared to sending directly from the device.
When the server confirm is not received or the cloud connection is not working, the device sends the data directly to the receiving server once, skipping the particle cloud.
I expect the slightly higher data consumption from this, is worth securing transfers regardless of how often and how long the particle cloud is out of service (almost).
I will give this a spin, and wonder what pitfalls this approach might have?
I think I follow what you are talking about but I am not sure how you will execute it. During an outage, there are a few different levels of failure:
- Carrier - the device will fail flashing green and not make a connection to the cellular network.
- Particle platform (better name?) - the device will fail rapid flashing cyan as the device attempts to log in. This was the outage of July 15th.
- Particle services - In this outage, the Particle device connects (breathing cyan) but Webhooks are not working (Have seen this as well).
I am not sure how you can send data anywhere in the first two scenarios and building the logic to handle these (and others I have not thought of ) will be hard.
I am going to pursue a couple different approaches:
- Make sure my devices don’t drain the battery trying to connect during an outage - I think I have this one nailed
- For short outages, @rickkas7 publishQueueAsync will buffer Particle.publish() for Webhooks
- For longer ones, I can put my data payload into an object and store it in FRAM. Then, when service is restored, I can send the data payloads with their timestamps. Ubidots will then put them into my device data stream in the correct order as it nothing happened.
What do you think?
Chip
1 Like
Hi folks,
I wanted to reach out here to let everyone know that Particle has been monitoring this discussion and that Particle is currently investigating concerns around customer inquiries regarding data consumption and bills.
It has come to our attention that a subset of self-service users on the Particle platform were not being charged for data overages for devices consuming above 3 MB of data. We also have identified a subset of customers whose devices may have seen data increases as a result of our recent incident on 7/15. We are in the process of identifying and reaching out to these customers. To try and address any questions in the meantime, please see below:
Why has my bill gone up?
Particle has identified an issue in our platform where, in recent months, a subset of users were not being billed for data overages when a device consumed over 3 MB of data. We have corrected this platform issue and customers should expect these overage charges to be reflected in their bills moving forward.
Why are my devices suddenly getting capped?
In addition to the above investigation, it was found that a subset of SIMs were not being capped upon reaching their data limits and were able to consume data far beyond their intended cap. This has also been corrected for these SIMs, and customers with devices consuming above their set cap will now see their devices being paused moving forward.
2 Likes
In the Carrier scenario upload is blocked. Unless it is actually the modem, and that can be tested with a reset and a Google DNS ping/dns look-up or similar.
In the other two Particle scenarios mentioned, the device builds a [http post] itself (HTTPclient Library or similar) and send it directly to the end service (e.g. Ubidots type of service), waiting with timeout for a confirm directly back. The increased data rate due to headers etc. will be acceptable the few times a year the Particle cloud is not operational (or if hypothetically closed).
If the end service (e.g. Ubidots type of service) is down, there is not much to be done at upload time, for concepts where the upload can not wait.
The Carrier or typical end service, is usually down very briefly and customers understand that. The Particle cloud has been multiple hours, days or even weeks, seen from the customer point of view, when devices needs to be power cycled in the field to re-enable normal operation. Customers I am aware of do not accept that at all. Hence the pursuit of having a fallback during cloud outages.
Good for scenarios where the upload can wait, but with option 3. available, I would skip option 2. (even if it is a very nice concept 
). For less complexity as 3. covers both short and long outages.
Options 3. could also work with direct upload to the end service mentioned above, if the outage turned out to be even longer.
as far as having to have a customer go reset a device we have not that problem since a long time ago with the eseries boards. As of now (knock on wood) our photons, argons, borons are up “all” the time.