Hi—I’m reading as much as I can find on this forum for storage of data that the Electron collects offline but still fairly uncertain where to start. I’m hoping to add a good chunk of external storage (~ 1 GB) and log everything then use the controller to decide what actually gets sent to the cloud. It looks like the flashee-eeprom library is still among the more commonly used libraries for external storage but was written for spark; is it still the way to go for solid state memory? I’d prefer not to use SD cards as this will be for a consumer product and I want everything to be board-mounted.
And, while perhaps it will be a little more expensive, will it even be possible to send 10s to 100s of KB to the cloud at once? I know I can send only 255 B in a given Publish( ) but will it be feasible to throw that in a loop until the Electron completes sending a file?
BTW—I called and asked the sales team these questions and they were unsure. I bought and am testing an Electron so any leads will be much appreciated! Thanks!
@ScruffR, though an embedded SD standard was developed, there are very few, if any, chips available on the market. It may be best to a) reduce the amount of storage required by processing the raw stream and/or b) use a microSD card with a captive (locking hinged lid) socket. Why microSD? The SDFat library supports DMA, is fast and reliable. Wear leveling is done at the device level, not in your code.
@supscientist, Particle.publish() is designed to send 255 ASCII characters per publish. Assuming your data is binary, if you convert binary to Base64, you could get about 170 binary bytes per payload. With 100KB to send, that would take 588 publish events. At one publish per second (unless you pay for Enterprise rates), that will take about 10 minutes to transmit. Basically 10mins/100KB.
The other alternative is to stream non-encrypted UDP or TCP data to a server directly. There is a HTTPS library available though I can’t comment on how suitable it would be in this case.
Thanks, @ScruffR and @peekay123. I haven't come across any good options for embedded SD but would love to learn about them if you know of any!
I have to pot the whole device in epoxy in a final predeployment waterproofing step. I'm concerned that SD card to SD receptacle contacts could be filled with insulating epoxy and break the electrical connection. Something I could fully solder on is desirable.
Regarding data transfer, I'll look into sending data directly to a server using the HTTPS and I will also hopefully get into the Enterprise program before long too. 10 min is too long—hopefully we can find a good workaround.
@supscientist, if you limit your storage, you could look at using SPI NOR Flash chips with SPIFFS. SPIFFS is not designed for large storage but could handle 128MB to 256MB storage devices.
Thanks again, @peekay123. SPIFFS looks promising; I’ll give that a shot. Though when I begin importing extra libraries like that, do I have to start using my own toolchain to build/compile projects?
I’m reading this thread (Suggestions for using external flash SPI) which looks particularly beneficial for my application. From your post here, it looks like SDFat still has some major advantages. If you happen across any surface mount components that work with SDFat, I’d love to learn about them.
Ultimately, I’ll want to be able to write around 100 MB (maybe more but I’ll stop there for now to keep things inline with SPIFFS capabilities) to some chip during an underwater deployment, then do some post-processing on the Electron then send post-processed data via cellular.
Do you think that Electron (or, perhaps, the incoming e-series) is well suited for this sort of application?
Cool—thanks! Intriguing, but most posts I've seen on that idea say there are lots of problems associated with various approaches to soldering a plastic-encased chip like that. Especially for what I hope will be a mass-produced item. Good to know that people have made it work though!
“Expandable flash storage for sensor data and/or backup application firmware”
It also says there’s 1MB Flash Storage but, surely this isn’t enough for sensor data… does anyone have any more details about this expandable flash storage as this might be the answer to the op
It would be interesting to find out which one, so we can build a breakout board for the Electron. This way prototyping on the Electron and moving over to the E Series will be much simpler.
You can get I2C EEPROM up to at least 32 MBytes on an 8 pin chip (pretty cheap, too). If you can get by with less than 1 GB, a couple of these chips strung together on one I2C bus might be a good solution. Unlike libraries that simulate EEPROM in flash memory, this is real EEPROM that does not need any wear leveling code to keep the memory cells from wearing out (albeit EEPROM will eventually wear out after 100,000 - 1 million writes). We successfully used EEPROM to persistently store system configuration and registration data on a Photon/Electron project where the simulated EEPROM did not work (I believe because of RAM limitations incurred for Flash wear leveling using the libraries). You can see what we did at:
