If you’ve been glued to the latest announcements from Amazon Web Services coming out of this year’s re:Invent conference in Las Vegas, then you may have seen Amazon’s most recent product introduction – Kinesis.
Kinesis is a low latency real-time data streaming and aggregation service for developers of applications and services that utilise or enable real-time data and analytics .
As part of the introduction of Kinesis, we at GoSquared have been working with Amazon to showcase the technology in the best way possible.
The Spec
The challenge – to come up with a beautiful, captivating demonstration of Kinesis’s abilities that will showcase the best web technologies available to the 8,000+ attendance of AWS re:Invent, and the 10,000+ watching around the world via the live stream video feed.
After Effects is for wimps
Let’s make one thing clear here – it’s called Amazon Web Services for a reason. So we felt very strongly that to do this demo right, we should do everything in the browser. Every single element of the Kinesis demo is built with open web technologies, with real data. There are absolutely no video effects software or post-production wizardry being applied here. All of this demo can be opened up in a Chrome browser or three.
3 Screens. 12,441,600 pixels
Talk about responsive web design. The audio visual setup at Amazon re:Invent is staggering. With 3 of the largest and highest resolution screens we’ve ever seen, we wanted to push the boundaries and take advantage of each individually to show the biggest, most daring visualisation we could. In order to have the best impact, we didn’t want to waste a single pixel.
Screen 1 – The Globe
3D globes are a speciality here at GoSquared. We wanted to showcase Kinesis’s ability to handle huge volumes of data, so what better data source than the Twitter firehose? The globe shows a portion of all Tweets being posted in real-time, firing out of their respective locations around the world. That’s right – each flying dot coming out of the globe represents a specific Tweet, and flies out from its respective location.
Each Tweet, or flying dot, is then sucked into the Kinesis magic box on the middle screen.
Screen 2 – Kinesis in action
We didn’t just want to show a random volume of Tweets flying into Kinesis on the middle screen. That’s why we synced up the left and the middle screen with WebSockets, so every single dot that flied out of the globe makes its way over to the middle screen and gets sucked into Kinesis without fail.
Out of Kinesis, the Twitter information is then passed into each of the Kinesis enabled applications running in EC2. As the demo progresses, more Kinesis apps are spun up to achieve the different tasks needed.
Notice the Kinesis magic box pulsating as Tweets fly in and are queued in real-time. Just to clarify – there were no images used in the making of the Kinesis magic box. In fact, the only images in the entire demo are the Amazon Web Services icons and the texture of the 3D globe.
Screen 3 – What you can do with the right tools
A lot of the focus of the demo is on the 3rd screen, on the right side of the stage. This is where we see how Amazon’s diverse range of web services can enable you to make discoveries within large data sets – if you pick the right tool for the job.
DynamoDB
To begin with, DynamoDB is used to produce a tag cloud communicating the most popular words and terms being Tweeted around the world in real-time. The tag cloud is then filtered down to just technology focused Tweets, and finally we focus in on terms related to space and the planets of the solar system.
Redshift
Next, we jump into the planet data, and turn our attention to data warehousing with Redshift, for running ad-hoc queries on historical datasets. We look at all Tweets in the last few days that mention “Mars”, and investigate the top other terms in those Tweets to give a little more context around what those Tweets are about. It turns out all these Tweets about Mars are not entirely focused on the planet, but about the singer – Bruno Mars!
RDS
Now that we have found this revelation, it’s time to jump into Bruno Mars focused Tweets and understand where they’re coming from. For this, we wanted to visualise another newly released Amazon Web Service, PostgreSQL on RDS. We looked up all Tweets in the US, and then zoned in on the west coast to highlight just how many Bruno Mars fans exist in and around the area.
It’s showtime for real-time
It’s been an honour to visualise the power of the tools that some of the smartest people in the world are building at Amazon Web Services. We hope you enjoy the demo, and we wish to thank everyone involved in making this happen – from the engineers on so many different AWS teams, to the execs, to the AV team, to Khawaja for getting up on stage in front of 8,000+ people.
