Exposing Data to Cloud - Project Architecture

Good afternoon!

To start, this is more of an architecture question rather than a technical one. I’m at the beginning of the concept phase for the electronics portion of a renewable energy project, and am looking for advice on exposing data from several tens to several hundreds of devices (Photons or Electrons) to the Cloud.

As a little background, this project focuses on the implementation of a kinetic energy harvesting flooring system which converts the wasted kinetic energy from pedestrian and vehicular traffic into usable electricity. The aim is to tap human motion in high traffic environments as a viable renewable energy source. Applications include proximity lighting, signage and way-finding, street lighting, mobile charging, security, etc. Currently, the flooring system takes the form of individual tiles which, when stepped on and depressed a few millimeters, drive an axial flux generator to generate electricity (renderings below).

The assembled tile. For scale: 12" x 12" x 2.5"


Axial flux generator. For scale: 5" x 1.5"


At the moment, the components for the energy harvesting mechanism, generator, and tile housing are being machined, so I’m focusing on the electronics. Ideally I’d like to have a Photon/Electron/P0 in each of the tiles, continuously monitoring power output of the generator, along with taking readings from an accelerometer to count footsteps/impacts. My concern is that, in order to get near real-time data, the microcontrollers would need to be POSTing very frequently. Given that several tens to several hundreds of tiles could be in close proximity, does anyone have advice/suggestions on getting data from each of the tiles to the Cloud? I am considering doing batch POSTs where one device polls the data of adjacent tiles every few seconds, notifies a server that there’s new data, and then the server GETs the aggregated data from the single device. However, that creates very large payloads which would require multiple requests.

I apologize for the long post; I wanted to give as much background as necessary to understand the problem. Again this is more of an architecture question for future scalability. Any thoughts/advice will be greatly appreciated.

Thanks!

1 Like

Fascinating concept!

In the next few weeks, we will be exposing SoftAP mode on the photon and allowing applications to use that as a network when creating sockets. This allows you to build a local network of cores/photons that connect to a designated Photon rather than having to connect via a central WiFI access point.

With this in place, you could designate some photons as aggregators that collate the data from neighbouring devices, and then forward the data to the cloud. The size of publish event data has recently been increased from 63 to 255 characters, so a compact format could allow quite a lot of information to be transmitted each second.

What I’ve described above will be available in a couple of weeks. It will require writing custom TCP clients and servers to communicate between the local devices and their designated aggregator device.

Looking forward to the future, there will be local publish and subscribe, so that it’s much easier to communicate information between local devices using the familiar Spark.publish and Spark.subscribe functions - your devices publish sensor data via Spark.publish() and the local aggregator subscribes to these notifications via Spark.subscribe() and collates the information as it’s received.

The aggregator could also be a photon coupled with an electron to bridge from local wifi to cellular networking.

This is of course just high-level overview, happy to go into more detail where needed!

1 Like

Thank you so much for the information @mdma! This is definitely what I was looking for.

The increase in publish event size is certainly a nice improvement. I’m looking forward to fitting more descriptive data in each publish.

My Photons are scheduled to arrive later this week, so after some experimentation, I’ll post back with questions.

Thanks again!

1 Like

Super cool project! I’m really looking forward to seeing more about this! :slight_smile:

1 Like

Thanks, @Dave! That’s really appreciated! I’ll be sure to post back soon with updates. :smile:

1 Like

I would forego the idea of a photon per tile, that seems like a scalability problem, I would encourage you to think of connecting groups of tiles together with something cheap and reliable, like CAN, MODBUS, or vanilla RS422/485, then equipping some “master” tiles with a photon.

It will save you power, cost and heartburn.

1 Like

That’s been my line of thinking over the last few days. It’d be quite expensive from a BOM standpoint to have a Photon per tile, and if this were to scale it might be difficult to maintain/update. I’ve been considering I2C between designated sub-groups — the Trekkie in me wants to call them Unimatrices — as tiles will be hardwired into groups for power sharing. For redundancy, if the I2C master or aggregator unit goes down in each sub-group, I’d like the network to reorganize so data isn’t lost. A wired approach seems most reliable so I’ll read up CAN, MODBUS, and RS-422/485 implementations. Thanks @AndyW!

1 Like

Don’t use i2c, you will want to use some differential signalling, not single-ended that can fall foul of differing ground references because of current flowing in the ground and with lousy noise immunity.

Ok. Thanks for the advice.

@gaudsend, I think IEEE 802.15.4-based wireless mesh network would be a good fit for this sort of application. The sensing tiles would be wireless nodes that form a robust, low-power mesh network. The data traffic would be routed automatically via neighboring nodes into a gateway device that aggregates the information to send to the cloud. I'm currently working on this project:

I've also developed a small (2 sq in) energy harvesting board that converts low level electrical input (100mV - 5.5V) from energy harvesting converters of different types (eg. solar, thermal, wind, motion, EM fields) at high efficiency (up to 93% using configurable MPPT) and stores the energy in rechargeable batteries or supercapacitors. It can power the mesh radios and other low-power sensors. It also can measure the level of input voltage from the energy harvesting converter. If you are interested, I can tell you more about it.