Best framework for data logging

I’ve been using Graphite/Carbon + StatsD for a while. It stores the metrics to disk. The graphing and calculation capabilities of Graphite are utterly amazing. I’ve even written a simple spark-carbon-library to push metrics from your Core/Photon directly into Carbon. (I should probably do one for StatsD as well).

The StatsD part is optional (but still available). The main thing I like about StatsD is that it can do “gauges” that will store the last known value of a metric and submit it every X seconds to Carbon so that you can have a continuous line graph instead of a bunch of dots.

Another option might be a Redis database. Someone wrote a Redis library a while back, but it hasn’t been ported for the web IDE. I haven’t used it personally, so I can’t offer much feedback beyond that.

I tried InfluxDB in the past and wasn’t pleased of how well it didn’t scale when it had tens (or was it hundreds) of thousands of records. I probably wasn’t doing any sort of proper roll-up of old stats. My opinion here may be tainted due to my own lack of effort. :wink:

There’s also SparkFun’s Phant to store data. There are a few different libraries available in the web IDE that may help: PietteTech_Phant, phant, Spark_Phant_Library. I haven’t used Phant or the libraries, so some extra leg work may need to be done to evaluate all of the options.

I’m personally planning on sending all my metric data from my Cores/Photons using Spark.publish() and write a listener/relay in Node.JS to watch for those messages and relay them into Graphite. That way, whenever I add a new device, there will be zero additional configuration on the server-side to pick up new metrics. And maybe I can write up a plug-in for Freeboard that will load historical data and plot new incoming metrics in real-time.

I’ve been researching this stuff off and on for a while and still haven’t found an easy solution that satisfies all my needs (fast, cheap, high-resolution, and no hard caps on metrics or storage).

5 Likes