Data-logging to remote server

I want to data-log sensor infornation to a remote server for storage (probably a mySQL database), which can then be accessed via a standard web page and charted using something like High Charts by a number of people.

I’m good with hardware and firmware, but my software for web needs a lot of improvement! Hence I’m a bit stuck as to how I’d go about this - ideally the core would post into the database rather than the server pulling the info using something like a cron job.

Any ideas and assistance greatly appreciated.

Have you seen @kareem613 's atomiot service:

Atomiot (formerly Spark Tools) Cloud services for your Spark Projects - Cloud - Particle

There is also a way to log to a Google spreadsheet:

Thanks for the prompt response.

I’d been down the Google route which although it worked, I found very sporadic in when and how often the script ran (I’d set it to 15mins and sometimes wouldn’t grab a sample for an hour or two).

I think I need something similar to the Atomiat service, but just not as a developer tool but running on its own webpage and without the need to enter access code or device ID

I also like the idea of push rather than poll. Cron jobs and other schedulers are just additional overhead to me.

If you don’t want to build your own solution, you can use a service such as Xively (formerly Cosm formerly Pachube) or ThingSpeak.

Let us know if you need any help!

Why wouldn’t Spark.publish() be the right solution in this case? Can’t I push data back into the cloud in a format that I could then parse into a database under my own control?

Or am I misunderstanding what Spark.publish() is all about?

Spark.publish() requires something to be listening to the server-sent events (SSE) stream. In my opinion, this is best used for updates that are not in regular intervals for polling or pushing. I think it’s actually harder to write something to monitor the SSE stream than it is to write a quick script and cron job for polling. Workers will be available soon that will allow for simpler webhooks similar to the HTTP client, but without all the hassle of using TCPClient. It also has the added benefit of supporting HTTPS (which a lot of services require these days).

Ah, I think I understand now. Thanks for explaining that.

@wgbartley I’ve as good as got this set up on a cron job, but you mentioned schedulers - is there an alternative to the cron job?

In the Windows world, they call it "Task Scheduler". I'm sure there are plenty of scheduler programs for all the operating systems, but cron is the only one that I can think of off the top of my head for *nix systems.

I think the essence of NanoAkron’s question is very important and, in fact, strikes at the heart of spark’s fundamental market and, probably, strategic plan. Thus, although the responses to NanoAkron’s question are very good and certainly welcome, I suggest it simply isn’t enough. In my specific case, I’ve been programming for over 50 years and earned a master’s in computer science, I am not (yet) a webhead and find that the constant use of acronyms and reference to strange programs to be not very helpful. I am not suggesting that Spark is somehow responsible to clearly explain the thicket of gobbledygook that has grown up around the interweb, I am suggesting that to do so in the limited context of the spark-core.is essential to realizing a significant chunk of the early IoT market.

In my specific instance, I got involved with the Spark-Core in contemplation of deploying a series of 30 core bi-directional networks. After 6 weeks of experience and reading and running example and tutorial code, I still don’t have an inkling what software technique to use although I do know many things which really scare me including reliability, restarts, timeouts, and the cloud. I suggest a carefully considered white-paper is in order.

Thank you for your reply captn.

This is the heart of it isn’t it - usability. I switched from PICs to the Arduino because of its IDE. I’ve developed a couple of neat things on PICs, but at the end of the day I was willing to exchange power and versatility for a solid IDE with a large user base that talked in public about the issues they were having.

Make the Spark IDE local, cross-platform and easy to use and I promise it will explode.

Lock it down, control the cloud and make bidirectional communication difficult and people will just wait for the competitors to catch up.

I want this to succeed - there are so many potential businesses that can be built from this.

Hey guys - just wanted to chime in. Great feedback from @captnkrunch and @NanoAkron both. There are a bunch of issues wrapped up here, so let me address them head on.

Documentation. There are about a million ways to use the Spark Core. Some of them are easy, some of them are more challenging. This is the nature of any open source project; because it’s not locked down, there are a million possible answers to any one question. There are absolutely holes in our documentation that cause the learning curve to be jagged instead of smooth, and we are beginning to work to close those holes, although it will take a little while. So please be patient, and know that we are very aware that there is a documentation gap.

IDE. I agree that there should be a great local development option, but I’m not sure our web IDE is it. The web IDE is designed for simple projects, and my hope long-term is that we smooth the path for popular existing local IDEs (such as Sublime Text or Eclipse) so that people can exploit the full power of the Spark Core once they cross a certain knowledge threshold and decide to dig in. However that said, I’m not at all certain on this path, and we’re still learning about how people use the IDE to figure out what the best local development path should be.

The Cloud. We will be publishing the open source version of the Cloud this summer. Don’t worry, it’s coming. We wanted to make the Cloud feature-complete before we focused on open sourcing it, and while we’re not quite there, we’re very close.

Don’t forget - this product is only six months old! It took Arduino years to hit 1.0. Keep in mind that it is still very early in our journey and the system we’re trying to build is quite large, and we are constantly juggling resources on building features, improving stability, creating documentation, and supporting the community. I simply request either patience or pull requests :slight_smile:

4 Likes

Thanks zach - glad to hear you’re aware of these issues and to hear how far you’ve come in such a short time.

Hope you can keep going at the same pace :smile:

3 Likes

@zach, a last thought - have you considered releasing an interim cross-platform IDE in the form of Arduino’s Processing-based IDE?

1 Like

I’ve definitely considered it, and I’ve also considered turning our web IDE into a cross-platform application using something like node-webkit. Also, the recently released atom IDE from Github could be an interesting tool to extend into a cross-platform application of our own.

What would you prefer? Personally, I am not a huge fan of the Arduino IDE. I love that it simplifies development, but I find its very poor management of libraries to be a big missed opportunity. That was the thought that led directly to the library management tool that we recently released and will be putting a lot of effort into improving.

One interesting datapoint is that Arduino is moving to a web-based IDE for Arduino v2.0; I’ll be very curious to see how that evolves.

1 Like

Thanks for the reply but I see I didn’t make my point very well. Yes the documentation needs work but things are improving rapidly, yes the IDE is primitive but in my experience (except for the lack of a monitor feature) it is very workable. Your webhead customers seem capable to create their own tool chains using their favorite components. I pointed to the cloud as a problem not because I don’t control it but because I don’t want it. If it makes development easier then fine but I have a website that effectively runs my business but I don’t maintain the servers… I pay someone to handle that headache.

But my main point is as you describe … the core is very flexible and, when combined with the interweb’s thicket of applications and tools literally millions of possible solutions potentially emerge. Therein lies your problem. I am sympathetic, honest. I run a business that allows non-professionals to use large CNC machines like lasers, mills, 3D printers, and plasma cutters. New members always ask what CAD program is required to control the tools. We could respond with a list of hundreds of available programs but instead we ask if the one they know can export .DXF files. If they don’t know, we recommend a (one) free CAD program and a tutorial to learn it.

My read of the IoT actually involves a relatively small number of basic system configurations … mostly small piles of cores talking bi-directionally to cell phones, large servers with databases, or cores. Are there other configurations? Of course but why not treat the most probable case for special effort? I suggest that Simplification is the key to acceptance of new technologies. Please don’t get bogged down with billions and billions…

1 Like

@zach

Thanks for your reply Zach. I’m going to make a similar point to @captnkrunch here - it’s simplicity that’s required for 99% of users.

I’m developing medical sensors and when I want to talk to a new DAC from TI, I fire up my Arduino IDE and just start writing. I don’t need to point to library directories, I just start writing and see what comes out. More than 99% of the time this is fine.

Occasionally I’ll need to directly access a port register for direct pin toggling or more time-sensitive stuff, but that’s a rarity.

Which aspects of the Arduino’s library management do you not like? For most users and most applications, going beyond a built-in library isn’t even an issue. But if you need to, good automatic installation tools (which the Arduino IDE doesn’t have) would more than compensate.

I have a device and a business in mind where I need direct control over the data flow to and from a remote wifi-connected device and a server which I control. The remote device deep-sleeps for most of the time, but wakes occasionally to call home, or if new data comes in it wakes up to send it back. That’s it!

Give me the tools to make that device and run it from whatever back-end I choose to implement, and you’ll have 1000s lining up behind me to do exactly the same with their businesses and ideas.

I know that making something look effortless is a lot harder than most people realise. But, that’s where the the greatest reward lies.

1 Like

I met the director from the eclipse foundation and we spoke about the spark platform.
He mentioned he wanted to speak with somebody at spark about using the Orion web IDE. Have you looked into this one? It looks amazing.

1 Like

Cool! Could you put me in touch? I’d love to chat with him, Orion looks pretty interesting.

2 Likes