We are looking into to use a Photon to log machine data every second (~200-500 Photon units). Is that kind of usage ok with Photon or Electron?
How/where are you planning to log it?
SSE are rate limited to 1p/s, so that should be okay. Using your own http stuff, there shouldnāt be a limit. Keep in mind that data on the electron is precious and sparse. Publishing every second is going to wear down the data rather quickly.
1 Like
Thanks, we are a company that works with āOverall Equipment Effectivenessā and we are looking into upgrading our hardware to collet the information we need. One problem we have is to get the data form the machines to the cloud. The customer IT department is really hard to work with so a GSM solution would solv this problem. We have our own HTTP endpoint that will collect and store the data.
SSE?
That sounds like a valid use of the Particle products 
With the HTTP endpoint youāve got, you can post the data to it using an http library on the device, thatās limited to how quickly you can call them and network latency.
Cellular is very interesting as well, since you then have no requirements for the network. The downside is that youāll consume a lot of data at the rate you want to publish. Depending on how you implement it, and how many calls you actually make, the data can be consumed rather quickly, making for an expensive collection. If thereās the option of using wifi, Iād definitely go with that. Might be a bit more bothersome to set up, but I think itāll pay off in the long term. Again, that depends on how you want to use it.
Do you really need the 1second resolution?
SSE (Server sent events) are like tweets for machines. The cloud will publish the data, to which your server can subscribe. Might be less interesting since youāve got your own endpoint.
Yes, 1 sec will be our limitation for this product, we have customer that is logging faster. But then we are using a PLC to handel all that data.
This is a typical machine we have our device on. A injection molding machine, for every cycle the we collect all the data.
2 Likes
Nice project! The 1p/s seems fair enough, and it should be doable. The biggest āissueā is how to connect them I guess. With wifi youāve got to have access, and with cellular youāll have to get a matching data plan.
Iām not entirely sure what might be the best option here, so Iāve called in for some extra advise from Particle. Most of them are probably asleep now, so their response might take a bit. Letās see what they, or others, have to say. In the meantime, if youāve got any other questions, feel free to ask them
Just to throw in a question till Particle responds:
Would a "hybrid" solution work too?
Collecting data in high resolution (and possibly publish locally - WiFi or BT) and only publish publicly less often but bigger chunks.
I don't exactly know the design of the Electron dataplan but other providers "charge" in 1KB to 100KB blocks even if you only send 1 byte, going through 1MB in no time.
I don't know what kind of data you intend to publish, but if it is process information, I'd guess the customer should be interested in getting access to that info and might give its IT a nudge to support you.
@ScruffR, I agree on the collection and bulk sending for the Electron. Data collection will need to include a time stamp so samples can be correctly reassembled at the server end. The size of the payload may affect the ability for the Electron to send its data within the 1 second time frame so data queuing will address this. However, data may need to be stored locally if the connection is lost so FRAM/SD/NVRAM storage may be necessary. 
1 Like
I will check if the āhybridā version could work.
The data is just sensor data, no value for others. We have a API that allow the customer to get the RAW data, but first of all is it for the customer dashboard to se the health of the machine and manage production. The problem with the IT is that if we want to open a port. It could take days for them to authorize that, and another days to do it. If they want to do it at allā¦ So a GSM is not a must, only to help the customer work around the IT department.
There is RTC on Electron right? Get unix timestamp with milliseconds?
I will check the performance when i get my unit
This 1 sec time fram is rare, often it is around 5-30 sec cycle. But if we manage to get it to work width that 1sec. That would be excellent!
@fotoKrille, yes the Electron has an RTC which will sync with the cloud. The RTC library (built-n) supports Time.now() for epoch time in seconds or millis() for milliseconds since boot.
I believe the trick here will be to establish the GSM carrierās minimum chargeable data size (eg. 1KB or 100KB, etc.), collect data to fill a buffer of that size and send when full. Buffering (circular, double) would allow for asynchronous capture and sending.
How is the data captured? Is it raw (digital, analog) or bused (I2C, SPI, Serial, etc.)?
As for 500 Electrons sending data at the same time, since they are being sent to your servers and not the Particle Cloud then there are no rate bottlenecks (besides the carrier and internet). You may want to consider the Enterprise Dashboard to manage all Electrons as a fleet of devices making updates and monitoring WAY easier.
2 Likes
Alright so i can not have Unix Millisecond Timestamp?
Unix Timestamp: 1360013296
Unix Millisecond Timestamp: 1360013296123
With out rely on the counter from witch the unit stated?
The capture part is different for each company / machine, some is RAW and some we can use CAN BUS / Serial.
With Particle Cloud do you mean Particle.publish() function?
@fotoKrille, unless I misunderstand, Unix timestamp is Epoch time in seconds, which is provided by Time.now(). It does not, however, provide millisecond accuracy. I am poking the Particle folks for their thoughts on this.
You will need to understand the best/worst case data payload size requirements as they will affect your carrier costs and transmission times. Your timing is good since some very smart members are in the process of posting a new CAN bus library for the Photon (and Electron). How you capture other data and how much may affect your selection of the Electron.
As for the Particle Cloud, that functionality is represented by these functions:
https://docs.particle.io/reference/firmware/photon/#cloud-functions
Another thing to mention is thst even UNIX Epoch time had a long long datatype you would not get precise UTC time (as a GPS time source would provide) since Epoch timestamp neglects leap seconds. So you might have a number down to milliseconds but if a leap second was added in between you would be a full second off.
Additionally free running RTCs and even Particle.syncTime() would still be off by a few milliseconds due to network latency.
So the point being, how important is the ms precission in relation to UTC really? Or would a near enough compound of RTC and millis() in deed suffice?
?!?
Unix time is just the number of seconds since the UTC Epoch, if thereās a problem with leap seconds it is with the libraries that turn that into human-readable dates, not with a 64-bit counter.
Maybe thatās not germane to the discussion, but I firmly regard Unix timekeeping as one of those āit aināt broke, donāt fix itā areas.
[EDIT: itās been pointed out that Iām overreaching, specifically: https://en.wikipedia.org/wiki/Unix_time so things might not be as simple or straightforward as I made it.]
[Further EDIT: Iāve obviously lived way too long in the land of NTP-corrected unixen. This guy uses the word āmessā to describe the intersection of Unix time and leap seconds, I stand corrected, but now Iām so far off the original topic that Iām in Cliff Clavin territory (and not for the first or last time.)]
5 Likes
Yes that is true that UNIX Epoch is seconds. Why the milliseconds is impotent to us is to calculate how much time the machine ha lost such as under 24h period. And as for e.g a injection molding machine, if it has a cycle time of say 5.65 sec if it is of by ~150 ms something is wrong.
And that if the timer is of by 10 sec it not the problem is not the problem. It is more let say the consistent of the cycle time with X ms margin of error tolerance.
Sweet! 
I totally understand the desire for sub-second resolution & accuracy.
In modern unixen getimeofday() returns a timeval structure, with a seconds field that directly mirrors classic seconds-after-UTC-epoch, plus a microsecond field, to provide the higher resolution; but thatās just more unix stuff, and Iām getting way off-topic for the purposes of a photon/electron.
@fotoKrille, there is no hope in heck that any Electron monitoring a machine will be able to measure events down to milliseconds especially due to the overhead of communications (CAN, serial, etc)! After all, how would the Photon know when an injection moldingās machine cycle started or stopped down to the millisecond!
I would assume the machine itself measures and makes available these types of measurement which the Electron will relay to your server.
So if I understand correctly, the Electron would solicit these event messages every second, possibly process (to compress) and queue them for sending to your server at fixed intervals. Is this correct?
Pace Pulse from a photosensor. When i get that i know that the machine has close / start a cycle. After i get that pulse i log the value for that cycle.
I would use the Photon to calculate short-term and long-term mean and standard-deviation values for the cycle time and report these four values every (say) minute. If I found that I wanted a more detailed view of the data, I would also histogram the cycle-times on the Photon into (say) 16 bins and report that less often, like every hour or every day.
3 Likes