Applications Have a Pulse and Storages Need to Read it

Year 1682, Qing Dynasty, Forbidden City. Emperor Kangxi paces back and forth anxiously, awaiting his imperial doctor to break the silence that has already frozen the palace to subzero temperatures. Pinched between the doctor’s fingers is a red cotton string. Taut and snug, the string disappears into a translucent silk curtain that obscures the ailing concubine, the emperor’s favorite. The doctor is feeling the pulse through the minute vibration transmitted across the cotton string that braces the concubine’s wrist. “Your majesty. It is just a minor cold. And by the way, congratulations, my lord, her highness is three months pregnant already!”

As dramatic and fictitious as it might seem, there is some truth to the above scene. In both eastern and western medicine, a patient’s pulse is a vital indicator of his or her well-being. While western doctors view pulses primarily as a means to measure heart rate, Chinese medicine practitioners read much deeper into it and use it as a key reference for the state of the patient’s health. They would use the length, the depth, and the quality, among other subtle variations in a patient’s pulse to identify the illness and pinpoint the root cause all the way down to a particular organ. Essentially what they do is signal pattern recognition in its most ancient form.

So what does all this talk about pulse have anything to do with storage systems? Well, everything. In today’s IT industry, any self-respecting storage system, at least a half-baked one, needs to have the capability to monitor, analyze, and respond to the IO patterns generated by each application. The storage management layer would read the “pulse” of the application and, based on historical traffic data, reach at a diagnosis and determine what storage resources are needed for a given time frame, and then formulate a scheme to predict the application’s future behavior and take preventive measures.

The ability to adapt and predict can be likened to the skills of a seasoned bartender. A good bartender not only serves the drinks, but also knows during what time of the day a certain drink is in demand and prepares in advance for that rush; he would also know what a frequent customer is most likely to order and start preparing that cocktail the moment that customer walks into the bar. We expect a good storage system to do exactly that. Adaptive and preemptive.

When it comes to adaptive control and pre-allocation in the consumer retail business, look no further than Amazon. No one does it better. Amazon adopts big data analytics to understand customer behaviors and uses that information to recommend highly relevant products to each customer. They even go as far as pre-shipping products that an individual is most likely to order in the imminent future to the nearest warehouse in advance, maximizing customer satisfaction and corporate profit at the same time.

Now consider the amount of unstructured data Amazon has to wade through in order to do all this pulse reading of the consumer behavior. If that’s not big data analytics I don’t know what is. Compare to Amazon, a storage system only has highly structured IO traffic data to analyze, and the sheer volume of the data must be several orders of magnitude lower what Amazon is dealing with. Given the advancement of signal processing and machine learning technologies we have seen in the past few decades, there is absolutely no excuse that a storage system doesn’t incorporate certain flavor of traffic modeling and adaptive control in today’s marketplace.

Wait, that’s not all. Pardon my infomercial reference, but there really is more to traffic modeling than predicting the future. What if the application changes its usage patterns? What if the prediction is not spot-on? There has to be a way for the application to tell the storage system that it’s either doing a great job or a lousy one. Ah-ha! This is where Facebook comes to the rescue. That’s right, the “Like” button. Through a northbound API, the storage communicates with the application and allows it to hit a “Like” button, or conversely, a “Dislike” button, to express its degree of satisfaction. Based on the “review” from each application, a closed-loop feedback control logic can be implemented in the storage system to make sure that the storage performance always matches the application’s needs, in real time.

Applications and storages can now live happily ever after.