Visualizing Data for Cloud-Powered Applications

I’ve always been fascinated by how much information can be consumed through graphics as opposed to text. By providing the right combination of shapes, colors, and numbers you can learn a great deal more about a system in a much shorter time that a text version might offer. I believe that an important step to sanely managing large scale infrastructure lies in the ability to visually monitor and interact with various trends among the different aspects of your platform. Finding and understanding important trends is a task that can still really only be performed by humans which is why creating visual representations of them is so important.

One of my favorite role models for interactive data is the site mint.com. For those unfamilar, mint.com is a personal finance site that allows you to track your income and expenses. I am a huge fan of their charting capabilities–which are appropriately labeled ‘trends’. Utilizing their trends, I am able to navigate and gain insights into my finances in a tastefully interactive manner. I can aggregate, filter, drill-down, and compare my finances by time, category, merchant and price. Their trends allowed me to discover valuable details about my finances in an easy-to-understand format. While a few thousand financial records might not qualify as ‘big data’, mint.com can teach us a lot about the potential for consuming information visually.

Mint Screen Shot

Data visualization from Mint.com

Previously, I worked on a marketing campaign platform that managed one-way interactions between businesses and thousands of clients. One of the harder challenges at this position was designing ways to visualize the effectiveness of specific advertisement campaigns. This was difficult at the time due to the restrictions on data interaction and visualization imposed by the need to support older browsers. By removing these restrictions we can leverage modern browsers to create interactive, dynamic, and information dense visualizations of data, such as this (click the image to see dynamic visualization examples):

Click to see visualization samples

I believe that bringing data interactivity and visualization to cloud infrastructure will enable operations and developers to dig for deeper insight into their systems. Cloud-based infrastructure offers easy access to a ton of data points about your environment that were previously difficult to access.  Data from the cloud vendor APIs can be combined with data from additional sources such as nagios, application loggers, financial data, etc. to discover important correlations and causations.

I joined Stackdriver to help application developers and operations people discover the trends that they care about most. These most valuable trends might not be initially obvious, but by allowing for many different permutations of data aggregation and presentation, new and interesting trends can emerge. Trends that can only help to expand the value large scale software has to offer.

What kinds of trends are you most interested in seeing?

One thought on “Visualizing Data for Cloud-Powered Applications

  • I like a nice chart as much as the next person, but in many practical business applications, in order to get to the compelling graphic, one needs to work with table data and do select/filter/sort operations to generate the result set that feeds the pretty chart.

    This is especially true in exploratory work where one needs to do some hard work to uncover something interesting. Well defined problems like spend by category are standard enough to draw a colorful pie chart directly. But if the question is: “which are the top influences from 20 dimensions that affect the ROI on my advertising campaigns?” Well then, let look at a chunk of data and figure this out.

    In other words, rumors of the demise of the data grid are greatly exaggerated.