Federator - The Horizontal Storage Controller
Horizontal Storage Controller
Federator is a control-path-only software-defined storage controller that simplifies and automates storage discovery, abstraction, delivery, monitoring, and optimizatioin.

Federator storage federation technology enables multiple pools from a heterogeneous physical storage environment to work together in an abstracted layer as virtual pools for data storage, access, migration, and management. Federator recognizes the built-in functionalities of underlying storage resources and uses them to define storage on demand, matching the requirements of end users or individual applications.

Traffic Modeling Module (TMM)
If there were unlimited IT budgets, an enterprise could certainly deploy an all-solid-state storage infrastructure for its most I/O-demanding applications. Unfortunately, most corporations don't have that sort of luxury so some kind of storage-equivalent of statistical multiplexing is desperately needed. After all, flash is still nine times more expensive than spinning disks.

In a typical enterprise environment, storage usage of a particular application rarely stays constant over the course of a day or week. Rather, there are peaks and troughs and each application's storage needs vary widely at any given time of the day. Take VDI (Virtual Desktop Infrastructure) for example, storage I/O traffic peaks during the boot storm between 8 and 9AM. It peaks again when employees are ready to wrap up the day's work and shut down their virtual desktop sessions between 5 and 6PM. Meanwhile, the company's ERP applications require the highest storage performance during the day, say, between 10AM and 5PM.

Not only do applications require different storage bandwidth at different time frames, their priority levels vary as well. Intuitively, the storage hypervisor can allocate a larger SSD cache space for storages assigned to VDI applications during the peak times, and release the flash to be allocated for ERP storages between 10AM and 5PM. But, imagine an environment that involves many more applications than just VDI and ERP and they all share the same storage infrastructure, it is humanly impossible for a system administrator to analyze all the I/O usage patterns and manually adjust the storage compositions, if at all, to optimize the overall system performance. This process must be automated.

Elastic Resource Control (ERC)
Storage Composition
ProphetStor storage hypervisor Flexvisor can fully leverage the installed server-side solid state drives to dynamically change the storage I/O throughput for an application. Take the above mentioned VDI situation for example, the application sees the same, say, 5TB volume, presented by Flexvisor during the two sample time frames, 5AM-7AM and 8AM-9AM. But underneath, the storage is accelerated by entirely different amounts of SSD cache.
During the peak hours, a much larger portion of SSD is given to the VDI volume to handle the high I/O demands. When VDI traffic decreases, the extra SSD capacity can be given to another application, say, ERP that can benefit from the boosted performance.

Flexvisor can not only dynamically adjust the SSD/HDD ratio in a given provisioned volume, but also optimize the caching mechanism in the SSD. For example, “hot” data can be selectively kept on SSD to improve random read speeds, while random writes are always cached and serialized before they are written to the HDD. These functions are of vital importance to the general performance of a storage array and they can and should be easily incorporated into Flexvisor.

However, no matter how powerful and optimized a single array is, without Federator, the entire compute-storage environment cannot be optimized as a whole. Again, this is why ProphetStor must develop its own storage hypervisor so the entire federated storage system can act like a well-conducted orchestra.

Adaptive Control and Pre-Allocation
Major online retailers apply big data analytics to understand its customers' behaviors and uses that information to recommend highly relevant products to the customers or even go as far as pre-shipping products that the customers are most likely to purchase to the nearest warehouse in advance, maximizing customer satisfaction and corporate profit at the same time. By the same logic, ProphetStor developed a proprietary traffic-modeling algorithm (patent pending) that utilizes fuzzy neural network technologies to analyze the applications behaviors in storage usage. The difference is, mega retailers have to analyze a huge amount of unstructured data while Federator only has to analyze a much smaller set of structured network traffic data, making its prediction much more accurate than Amazon's. With Flexvisor, this crucial knowledge of the network usage patterns can then be used by Federator to intelligently and dynamically allocate, or pre-allocate, needed storage resources to areas that need it the most in real time.
Key Benefits
How it Works
When Federator is introduced into a storage environment, the system administrator needs to perform a few simple steps to include the desired storage arrays to be managed. The same routine has to be repeated every time when a new array is added to the network.
Federator automatically discovers supported storages on the network. Manual discovery is also supported.
Discovered storage resources and their associated features and characteristics are imported and abstracted into virtual storage pools.
Virtual storage pools are classified based on their capabilities and performance characteristics. Administrators can set an arbitrary cost value or any other custom tag for each storage pool.
Administrators create service catalogs with storage offerings based on different capabilities and performance levels such as size, IOPS, cost, and thin provisioning. For ease of use for the end users, administrators may choose to create a "menu" of pre-configured storage tiers for typical applications such as Oracle database, web server, or VDI.
Users can request for storage through a customized self-service portal such as OpenStack Dashboard. Applications can also request for storage directly through an HTTP REST API.
OpenStack Integration
Federator can be run in any cloud environment, and is already tightly integrated with OpenStack, the leading open-source cloud operating system. OpenStack users can access Federator directly from Horizon.
Federator features a comprehensive open RESTful API that allows software vendors to develop rich data service applications based on a flat storage infrastructure. Listed below are the supported classes:
  • Authentication
  • Sessions
  • Users
  • Tasks
  • Unmanaged
  • Orders
  • Offerings
  • Initiator Groups (igroups)
  • Initiators
  • Hypervisors
  • Hypervisor Portals
  • Hypervisor Adapters
  • Pools
  • Pool Profiles
  • Volume Groups (vgroups)
  • Volumes
  • Replications
  • Snapshots
  • Exports
  • Licenses
To show how easy it is for applications to programmatically request storage services, here is an example of a CURL command to request a clone based on an existing snapshot.
CURL Command
$ curl https://{root}/v2/volumes/ \
  -H "X-Auth-Token: fed_test_ed00058da30a048240f824" \
  -H "Content-Type: application/json" \
  -d '{
     "offering": 1,
     "capacity": 1099511627776,
     "alias": "MyVolume“,
     "from": "/v2/volumes/1/snapshots/2"
{   "id": 1,
    "uuid": "8b3d5c13-e858-4a21-9aa7-491770d40d3b",
    "vgroup": null,
    "alias": "MyVolume",
    "description": "",
    "capacity": 1099511627776,
    "state": "online",
    "type": "volume",
    "offering": 1,
    "pool": 1,
    "from": "/v2/volumes/1/snapshots/1",
    "snapshots": "/v2/volumes/1/snapshots/",
    "exports":  "/v2/volumes/1/exports/"