I’ve always been fascinated by how much information can be consumed through graphics as opposed to text. By providing the right combination of shapes, colors, and numbers you can learn a great deal more about a system in a much shorter time that a text version might offer. I believe that an important step to sanely managing large scale infrastructure lies in the ability to visually monitor and interact with various trends among the different aspects of your platform. Finding and understanding important trends is a task that can still really only be performed by humans which is why creating visual representations of them is so important.
One of my favorite role models for interactive data is the site mint.com. For those unfamilar, mint.com is a personal finance site that allows you to track your income and expenses. I am a huge fan of their charting capabilities–which are appropriately labeled ‘trends’. Utilizing their trends, I am able to navigate and gain insights into my finances in a tastefully interactive manner. I can aggregate, filter, drill-down, and compare my finances by time, category, merchant and price. Their trends allowed me to discover valuable details about my finances in an easy-to-understand format. While a few thousand financial records might not qualify as ‘big data’, mint.com can teach us a lot about the potential for consuming information visually.
Previously, I worked on a marketing campaign platform that managed one-way interactions between businesses and thousands of clients. One of the harder challenges at this position was designing ways to visualize the effectiveness of specific advertisement campaigns. This was difficult at the time due to the restrictions on data interaction and visualization imposed by the need to support older browsers. By removing these restrictions we can leverage modern browsers to create interactive, dynamic, and information dense visualizations of data, such as this (click the image to see dynamic visualization examples):
I believe that bringing data interactivity and visualization to cloud infrastructure will enable operations and developers to dig for deeper insight into their systems. Cloud-based infrastructure offers easy access to a ton of data points about your environment that were previously difficult to access. Data from the cloud vendor APIs can be combined with data from additional sources such as nagios, application loggers, financial data, etc. to discover important correlations and causations.
I joined Stackdriver to help application developers and operations people discover the trends that they care about most. These most valuable trends might not be initially obvious, but by allowing for many different permutations of data aggregation and presentation, new and interesting trends can emerge. Trends that can only help to expand the value large scale software has to offer.
What kinds of trends are you most interested in seeing?