Spark Publish Visualization

Came up with a pretty useless visualization for the public firehose, just thought I would share!

Backstory is, I was playing around with reading the public publish events, and I couldn’t find a decent way to get every event with EventSource (EventSource requires the name of the event to filter on). So, I created a node.js app that listens for the events from the public firehose and then turns them around to the client, but strips away the ‘event’ tag so the client can now use EventSource for all of them.

Just so I could verify it was working, I made the web page to visualize it. It really is pretty useless, but decided to throw it up on my home server anyway. Enjoy!

5 Likes

Hey @eely22 that’s really cool! ^^

This does not seem to do anything in my case (Tested on Win8.1 Chrome, Firefox, IE, Maxthon).

No circles seen here either…

UPDATE: Looking at the logs, after about an hour and a half, the events just stopped. I added some error handling that I hope will fix that in the future, not sure why it stopped, but I will try and keep it going this time!

Interesting, it seems like the node.js app stopped receiving events after some amount of time. I will have to look into that. I restarted it and it is working again, but I’ll keep an eye on it.

One thing, it probably won’t work on IE, which doesn’t support SSE. Chrome and Safari should be ok

1 Like

Now I see them… Fascinating!

I wanted to show a little more, so I added a second graph. The first shows only the first event from each Core, it builds up a picture quickly but won’t change much over time. I wanted to show every event.

The second graph now shows a “ping” for every public event published. The X-Axis position is based on the event length and the Y-Axis is based on the order in which we saw the Core.

Check back and scroll to the bottom to see the new graph. Again, mostly useless, but kind of mesmerizing…

That’s cool :+1:

It would also be cool to have some info about the event when hovering over a circle.

@ScruffR Agreed! I added info boxes so when you mouse over, you can see the Core ID and the length of the data. I also wrapped the Y-Axis of the Ping graph, otherwise if there were more then 64 Cores, the data would all be on the bottom axis.

It is pretty interesting to watch the graph actually. You can see how some Cores constantly send data every X seconds, some send multiple events simultaneously every X seconds, some are bursty, and some just seem random.

By the way, added this to github if anyone is interested in the code:

1 Like