Format

Send to

Choose Destination
Proc USENIX Annu Tech Conf. 2008 May 1;2008:391-404.

Free Factories: Unified Infrastructure for Data Intensive Web Services.

Abstract

We introduce the Free Factory, a platform for deploying data-intensive web services using small clusters of commodity hardware and free software. Independently administered virtual machines called Freegols give application developers the flexibility of a general purpose web server, along with access to distributed batch processing, cache and storage services. Each cluster exploits idle RAM and disk space for cache, and reserves disks in each node for high bandwidth storage. The batch processing service uses a variation of the MapReduce model. Virtualization allows every CPU in the cluster to participate in batch jobs. Each 48-node cluster can achieve 4-8 gigabytes per second of disk I/O. Our intent is to use multiple clusters to process hundreds of simultaneous requests on multi-hundred terabyte data sets. Currently, our applications achieve 1 gigabyte per second of I/O with 123 disks by scheduling batch jobs on two clusters, one of which is located in a remote data center.

PMID:
20514356
PMCID:
PMC2877279

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center