Resources

Building Impactful Data Visualisations

Steve Moreton summarises his thoughts after Influxdays

Steve Moreton - Senior Technical Director, CJC

I had the pleasure of speaking at the Influxdays this week and I’d like to thank Chris Churillo and her team at Influxdata for organising a wonderful event.

I was there to talk about how CJC have approached the task of building impactful data visualisations to be used by demanding capital markets firms for IT Operations Analytics (ITOA). Our platform, mosaicOA, uses InfluxDB to store metrics from hundreds of specialist servers, networking and middleware systems,.The data (CPU, memory, network and application data) is queried and visualised for clients for a variety of beneficial use cases such as root cause analysis, capacity management and machine learning.

But I also wanted to talk about ITOA and I thought it might be useful here to summarise my presentation into 4 statements

ITOA is a satnav of IT infrastructure

This analogy had been partly prompted by an incident three years ago when my colleague Peter Williams and I lost our way walking to a client. I say lost, but with over thirty years’ experience of working in the area, we were never going to necessarily admit to that (!) – so we reviewed the data available:

  • We knew where we were
  • We knew where we had come from
  • We knew the address of where we were going to
  • The exact path to get there had not yet been confirmed

So, strictly speaking, we decided we weren’t lost at all !! But this story did provide a helpful way of describing ITOA. It’s the modern and revolutionary methodology of supporting your IT infrastructure. Moving away from traditional real time monitoring, ITOA stores all your infrastructure data for continual analysis – ultimately becoming a satnav of IT infrastructure.

  • One server easily has 10.7 million changes (CPU changes etc) a day. Sixty servers can generate over one million changes a minute, so you don’t know where you are
  • If you don’t know where you are, you won’t have a record of where you have been
  • You can have a have an infrastructure goal or destination (cloud anyone?) but without the historical information, you can be making dramatically bad choices on what journey to make

 Cloud made ITOA, ITOA makes Cloud

ITOA had for a while been prohibitive, with many firms being put off by the high costs of database and physical infrastructure. However CJC moved ahead with the development of mosiacOA using cloud technology and open source software. 

From 2011 we were already providing services from AWS and in 2014 had established our own private cloud in Equinix.  We chose InfluxDB, an open-source time series database.  In 2015, we already had an early adopter Tier 1 North American Investment Bank connected and visualising their QA systems on fifty-five servers.

Clients on-boarded with mosaicOA are seeing huge benefits including:

  • Significant hardware footprint reduction, with applications moved to capacity free servers
  • Cloud Provider Transition – providing accurate due diligence on which provider will be best suited for specific application and importantly, who wouldn’t be.
  • Major hardware vulnerability patch testing 

ITOA and QA / Development is an unbeatable combination

The Intel Meltdown was a significant milestone for mosaicOA.  This was a major patch release which affected many industries and a quick timeframe was needed to test and ensure that the patch would not have had detrimental impact on systems.

We are increasingly finding that ITOA has given huge power to DevOps teams.  Retaining the data means that events can be ‘replayed and reviewed’ to do root cause analysis, anomaly and pattern detection. 

It’s the advent of ‘human machine learning’

Living in an age of self-driving cars, automatic assistance, machine learning and AI, suddenly we mortals, complete with human error and confirmation bias, start to look obsolete.   How many times has a vastly knowledgeable and experienced engineer had to theorise what could be causing an issue and in effect ‘guess’ the solution.

But when presented with rich and complete information, we can make better and quicker decisions.  MosaicOA has built in data transformation techniques such as principle component analysis (PCA).  The visualisations provided from mosaicOA provides we mortals with far better decision-making abilities. 

Steve

  • Post
  • Cloud
  • Technology