Cost effective, high performance solutions using erasure codes for big data management in large data centers

Sponsored by NSF CNS-CSR

Abstract:

Data and I/O availability is an increasing concern in today’s large data centers where both data volume and complexity are increasing dramatically. Most existing solutions are based on multi-replication techniques to provide data redundancy, where data chunks are replicated across storage server nodes. However, multi-replication techniques are insufficient to manage big data: it’s a big challenge to efficiently replicate N copies of a data set of tens-to-hundreds of petabytes! As an alternative solution, erasure codes tolerating multiple failures can provide reliability and availability at much lower cost. However, the biggest challenge using erasure codes to manage big data is the performance problem due to the complex encoding/decoding operations, which limits the application of erasure codes in large-scale data centers.

This project develops cost effective techniques to exploit erasure codes to achieve high availability and enhance performance in large data centers to efficiently manage big data via several research innovations. This project cohesively investigates how to utilize proper spatial cost and system/architecture techniques to improve the overall data access performance of server clusters built upon erasure codes. This research has fundamental contributions to pave the way to efficiently deploy data centers using erasure codes.  It has potential to benefit numerous big data applications such as online searching, social network, e-business, health care, and so on which are typically data intensive.

Personnel

- Principal Investigator

- Collaborators

- Post Doctoral Researcher

- Graduate Students

- Undergraduate Students

Recent Publications

  1. S. Li, Q. Cao, S. Wan, W. Zhang, C. Xie, X. He, and P. Subedi. "PPM: A Partitioned and Parallel Matrix Algorithm to Accelerate Encoding/Decoding Process of Asymmetric Parity Erasure Codes", Proceedings of the 44th International Conference on Parallel Processing (ICPP'15), Beijing, China, September 2015.
  2. P. Subedi, P. Huang, B. Young, and X. He. "FINGER: A Novel Erasure Coding Scheme Using Fine Granularity Blocks to Improve Hadoop Write and Update Performance", Proceedings of the 10th IEEE International Conference on Networking, Architecture, and Storage (NAS'15), Boston, USA, August 2015.
  3. M. Fu, D. Feng, Y. Hua, X. He, Z. Chen, J. Liu, W. Xia, F. Huang, and Q. Liu, “Reducing Fragmentation for In-line Deduplication Backup Storage via Exploiting Backup History and Cache Knowledge”, IEEE Transactions on Parallel and Distributed Systems, DOI: 10.1109/TPDS.2015.2410781, March 2015.
  4. Y. Tan, Z. Yan, D. Feng, X.  He, Q. Zou, and L. Yang, “De-Frag: an efficient scheme to improve deduplication performance via reducing data placement de-linearization”, Cluster Computing, Vo. 18, No. 1, March 2015.
  5. M. Fu and D. Feng and Y. Hua and X. He and Z. Chen and W. Xia and Y. Zhang and Y. Tan, “Design Tradeoffs for Data Deduplication Performance in Backup Workloads”, The 13th USENIX Conference on File and Storage Technologies (USENIX FAST), Santa Clara, CA, Feb. 16-19, 2015 (acceptance rate: 28/130=21.5%).
  6. P. Subedi, P. Huang, X. He, M. Zhang, and J. Han, "A Hybrid Erasure-Coded ECC Scheme to Improve Performance and Reliability of Solid State Drives", 33rd IEEE International Performance Computing and Communications Conference, Austin, TX, December 2014.
  7. M. Stuart, T. Lu, and X. He, "Alleviating I/O Interference via Caching and Rate-Controlled Prefetching without Degrading Migration Performance", 9th Prallel Data Storage Workshop (PDSW), held in conjunction with SC'14, New Orleans, November 2014.
  8. T. Lu and M. Stuart and K. Tang and X. He (2014). Clique Migration: Affinity Grouping of Virtual Machines for Inter-Cloud Live Migration. International Conference on Networking, Architecture, and Storage (NAS). Tianjin, China. Best Student Paper Award.
  9. S. Li and X. He and S. Wan and Y. Guo and P. Huang and D. Chen and Q. Cao and C. Xie (2014). Exploiting Decoding Computational Locality to Improve the I/O Performance of an XOR-coded Storage Cluster under Concurrent Failures. International Symposium on Reliable Distributed Systems (SRDS). Nara, Japan.
  10. P. Huang and P. Subedi and X. He and S. He and K. Zhou (2014). FlexECC: Partially Relaxing ECC of MLC SSD for Better Cache Performance. The USENIX Annual Technical Conference (ATC). Philadelphia.
  11. M. Fu and D. Feng and Y. Hua and X. He and Z. Chen and W. Xia and F. Huang and Q. Liu (2014). Accelerating Restore and Garbage Collection in Deduplication-based Backup Systems via Exploiting Historical Information. The USENIX Annual Technical Conference (ATC). Philadelphia.
  12. P. Huang and G. Wu and X. He and W. Xiao (2014). An Aggressive Worn-out Flash Block Management Scheme to Alleviate the SSD Performance Degradation. The European Conference on Computer Systems (Eurosys). Amsterdam, The Netherlands.
  13. Y. Yu, X. He, H. Guo, S. Zhong, Y. Wang, X. Chen, and W. Xiao, “APR: A Novel Parallel Repacking Algorithm for Efficient GPGPU Parallel Code Transformation”, Proceedings of Workshop on General Purpose Processing Using GPUs (GPU-7), 2014.
  14. Y. Yu, X. He, H. Guo, Y. Wang, and X. Chen, “A Credit-Based Load-Balance-Aware CTA Scheduling Optimization Scheme in GPGPU”, International Journal of Parallel Programming, August 2014.
  15. Guanying Wu, Ping Huang, and Xubin He, "Reducing SSD Access Latency via NAND Flash Program and Erase Suspension", Journal of Systems Arhcitecture, Vol. 60, No. 4, April 2014, pp. 345-356.
  16. C. Wu, X. He, Q. Cao, C. Xie, and S. Wan, "Hint-K: An Efficient Multi-level Cache Using K-step Hints", IEEE Transaction on Parallel and Distributed Systems, Vol. 25, No. 3, March 2014.
  17. Guanying Wu, Xubin He, Ningde Xie, Tong Zhang, "Exploiting Workload Dynamics to Improve SSD Read Latency via Differentiated Error Correction Codes", ACM Transactions on Design Automation of Electronic Systems (TODAES), Vol. 18, issue 4, October 2013.
  18. S. Wan, X. He, et al, “An Efficient Penalty-Aware Cache to Improve the Performance of Parity-Based Disk Arrays under Faulty Conditions”, IEEE Transactions on Parallel and Distributed Systems (TPDS), Vol. 24, No. 8,August 2013.
  19. Chentao Wu and Xubin He (2013). A Flexible Framework to Enhance RAID-6 Scalability via Exploiting the Similarities among MDS Codes. the 42nd International Conference on Parallel Processing (ICPP'2013). Lyon, France.
  20. X. Zhou, Q. Cao, C. Xie, and X. He, “D-PALD: A Dynamic Power-Aware Load Dispatcher with Response Time Percentile Guarantee in Heterogeneous Clusters”, Proc. of the 8th IEEE International Conference on Networking, Architecture, and Storage (NAS), July 2013 (acceptance rate: 37.5%).
  21. Hua Wang, Ping Huang, Shuang He, Ke Zhou, Chunhua Li, and Xubin He, “A Novel I/O Scheduler for SSD with Improved Performance and Lifetime”, Proc. of the 29th IEEE Symposium on Massive Storage Systems and Technologies (MSST), May 2013 (acceptance rate: 30 out of 109 submissions=27.5%).
  22. Pradeep Subedi and Xubin He, "A Comprehensive Analysis of XOR-based Erasure Codes Tolerating 3 or More Concurrent Failures", Proc. Of the 18th IEEE Workshop on Dependable Parallel, Distributed and Network-Centric Systems (IEEE DPDNS), in conjunction with IPDPS’2013, May 20-24, 2013.
  23. Tao Lu, Morgan Stuart, and Xubin He, "SLM: Synchronized Live Migration of Virtual Clusters across Data Centers", Poster presentation, the 11th USENIX Conference on File and Storage Technologies (FAST2013), Feb 12-15, 2013.

Software Release

Destor: An experimental platform for chunk-level data deduplication [USENIX FAST'2015]. This is a joint work with HUST collaborators M. Fu, D. Feng, Y. Hua, et al.

Sponsors