Data logging memory collection and dumping to a PC?

Hi guys,

I’m very new to this AND I am old. :grin:
I was using an Arduino Mega but I see so many benefits with the Photons.
I have a Photon - I got it working through persistence to blink the LED. YES!

So in transistion to Photon I ask -

If I monitor an analog input how many reads can I save into an array before I hit some sort of limit?

And

I want to transfer the array to a PC. Do I just use Putty for that (Like I was doing with the Mega) or what options are there in this modern world I find myself in?

Regards Dave

I too am old - seems most of get that way originally…

Monitoring an analog - depends on how many/how often.

You can use main memory - if you only want say 32k
You could use some frams - can store lots quickly.

You could use a sd card - fairly quick and holds lots of data…

If you are capturing slow enough, make a couple of buffers in main memory - say 1k each.

Use the s/w timers to capture the data into a buffer and switch to another buffer when full - have loop send full buffer via TCP/Ip to somewhere else.

So - whilst your questions appear on the surface to be simple (and good questions BTW), there are multiple ways of achieving your aims.

FWIW - here is how I would go.

Start by implimenting the timers (look for timer library). and capture into one of two buffers.
Have a simple print “Buffer X Full” when ever buffer gets full and switch to secondary buffer.

Once this is all working, try to write some dummy code that uses TCP/IP to send some data to another TCP/IP listener.

Once you have this working, merge the two programs together.

I may have some sample code that impliments double buffering (I think it stores to a SD card).

Stan

1 Like

Thanks Stan,

The data -

Simply I need to collect 2 analog signals every 0.001 seconds for about 10 seconds, so about 20,000 analog sample points. Easy enough I hope. Can the Photon store that?

Next I need to get these to my PC and finally into an Excel Spreadsheet for analysis.
I guess there is a modern tool like Putty for the cloud or a way to configure a web site to collect and store the data?

I’m old school Hyperterminal guy and more recently Arduino with Putty. I just have no idea how to get my data out.
So I might as well learn something new because technology moves so fast and keeping up is getting harder to do.

This is probably a little more complicated than what you need, but is an example of how you can capture data at high rates and send it via Wi-Fi to a computer on your local network. It’s for audio, but really it’s just reading an analog pin.

2 Likes

Rickkas7,

I will look at that ASAP. Thanks for the tips.

Dave

Hi Rickkas7,

I tried to verify your file but it does not like these library files

#include "Particle.h"
#include "SparkIntervalTimer/SparkIntervalTimer.h"

Is there a trick to getting these included in the code?

Thanks heaps, Dave

No trick. You need to add those libraries to the project, not just add in the include statements (in fact, those will be added for you when you include the libraries).

  1. Click the libraries icon in Build, and use the search to find the library you’re interested in.
  2. After selecting the library, click on the “INCLUDE IN APP” button
  3. Choose your app from the list that comes up
  4. Click on the “ADD TO THIS APP” button

This will add the library to your app, and automatically add the #include statement

1 Like

Ric,

You are awesome.

I will have a better look at the code tonight. :smile:
Dave

Ric,
I can almost understand that code.

I am away from my pc so I cannot be specific but what is happening on the pc side of the opperation? Is that something you wrote or are there heaps of these to choose from?

regards Dave

Not sure what you’re asking about. What code? What “heaps of these” are you referring to?

Ric, It was in reply to rickkas7. :smile:

Dave

I’ll have another tutorial with a bunch of code examples for sending data from a Photon to a computer, probably later this week.

But I wrote the code on both side of the photon audio sample code, so if there’s something specific you don’t understand I can probably explain what it does.

3 Likes

Thanks. Working through it slowly.

Dave

Hi,

I like Rickkas7 idea of writing files but –

Just trying to get my head around the web and publishing feature. If I publish data to https://console.particle.io/logs how can I locate or link this data in something like excel?

Dave

Rickkass7,

// This is the IP Address and port that the audioServer.js node server is running on.
IPAddress serverAddr = IPAddress(192,168,2,4);
int serverPort = 7123;

What is happening in audioServer.js ? Is it something you wrote or are there heaps of these in a library?

I am assuming your PC at 192,168,2,4 (For anyone else new to this look at the devices via your Router or phone wifi settings)
But how did you discover the value 7123?

What is the difference between sampleBufs and SampleBuf

Which command actually sends the data? And what does the data look like? Syntax and formatting? Space or comma etc?

Regards Dave

Yes, 192.168.2.4 is the IP address of the computer running node.js. The port number 7123 doesn’t matter, other than it can’t already be in use by another program on the computer. You can set it to something else as long as you change it both in the node.js code and in the Photon code.

The audioServer.js file runs in node.js on your computer. It listens on a TCP network port (7123) for connections from the Photon. In this example, the data is raw binary data (8-bit audio samples) and then writes them to a binary (wav) file.

The technique can easily be extended to do comma separated values, JSON, or whatever you want, though this example program doesn’t show how to do it.

The Photon sender uses double-buffering. Each buffer is a SampleBuf. The idea is that you fill up the buffer with samples until full, then you start filling the other buffer. The complete buffer is then sent over the network, which occurs at the same time as the other buffer is being filled, so the samples are obtained a constant rate regardless of how long the network transfer takes.

2 Likes

I am getting my head around this - thanks for the input.

From https://github.com/rickkas7/photonAudio/blob/master/audio3/audioserver.js


  writer.write(buf);
});
socket.on('end', function () {
  console.log('transmission complete, saved to ' + outPath);
  writer.end();

Increment the filename's new number
You set up a directory and filename,
Convert data to signed 16 bit
Write the data from the code above.
Close the file with Writer.end() ????
Listen to the dataport?


In windows command prompt, will I see the messages from console.log('text") ???? (or where will i see the message?)

console.log('transmission complete, saved to ' + outPath);

net.createServer(function (socket) {
  console.log('data connection started from ' + socket.remoteAddress);

I don't really understand this code?????
What is the first line doing? And what is a socket?
remoteAddress - where does this value come from?


Thanks in advance again :smile:
Dave

Sorry, I forgot to respond to this yesterday. Your assumptions for the first block of code are correct.

Since you run node.js in a terminal window like:

node audioserver.js

all of the console.log output will go to that terminal window.

The net.createServer code is just how node.js Javascript code is written. It’s weird at first, but you get used to it.

The syntax net.createServer(function (socket) is an anonymous inline function declaration. It’s roughly equivalent to code that would look sort of like this:

net.createServer(myServerHandler);

// some other code goes here ...

// Definition of myServerHandler
function myServerHandler(socket) {
    console.log('data connection started from ' + socket.remoteAddress);
    ...

In any case, the server handler function is called whenever a new connection comes in. It gets passed a parameter, which is a socket object, that includes information like the IP address and port of the initiator of the connection, as well as methods for reading and writing to the socket.

That’s defined by node.js, for example:
https://nodejs.org/docs/v4.4.7/api/net.html#net_class_net_socket