Untitled Document

Distributed Symmetric Active/Active Metadata Management and Bulk Data Transfer

Sponsored by NSF and ORNL

Abstract:

This project explores high availability data storage services through distributed symmetric active-active metadata management. High availability storage service is critical for scientific high-end computing (HEC) systems, in which there is a demand for continuous non-stop 24/7 availability. The research investigates a group of metadata servers in a symmetric active-active mode to collectively take on the responsibility of managing the system and its respective state information. It treats metadata separately in dedicated metadata servers and allows more than one redundant server to be active. Metadata state replication is performed by totally ordering all state change messages and reliably delivering them to all redundant active metadata servers. A group communication system is utilized to ensure total message order and reliable message delivery as well as metadata server membership management. This research provides a proof of concept for three key problems: global state identification and consistency, group communication overhead, and metadata server membership management.

In addition, we also investigate the issues to transfer bulk data over the network. We develop a protocol, Performance Adaptive UDP (henceforth PA-UDP), which aims to dynamically and autonomously maximize performance under different systems. A mathematical model and related algorithms are proposed to describe the theoretical basis behind effective buffer and CPU management. A novel delay-based rate throttling model is also demonstrated to be very accurate under diverse system latencies.

Personnel

- Principal Investigator

Dr. Xubin He, Tennessee Tech University, PI

- Collaborators

Dr. Stephen Scott, Oak Ridge National Lab
Dr. Chrstian Engelmann, Oak Ridge National Lab
Dr. Qishi Wu, University of Memphis
Dr. Box Leangsuksun, Louisiana Tech University
Yung-chin Fang, Dell Inc.
Changsheng Xie, Huazhong University of Science and Technology
Jizhong Han, Institute of Computing Technology, Chinese Academy of Sciences

- Graduate Students

Li Ou, Tennessee Tech University (PhD, graduated in 2006, is currently with Dell Inc)
Xin Chen, Tennessee Tech University (PhD student, 2007-)

- Undergraduate Student

Ben Eckart, Tennessee Tech University (2007-2008)

Recent Publications

Ben Eckart, Xubin He, Qishi Wu, and Changsheng Xie, “A Dynamic Performance-Based Flow Control Method for High-Speed Data Transfer”, To appear in IEEE Transactions on Parallel and Distributed Systems, 2009. Preprint: doi:10.1109/TPDS.2009.37
Xubin He, Li Ou, Christian Engelmann, Xin Chen, and Stephen Scott, “Symmetric Active/Active Metadata Service for High Availability Parallel File Systems,” Journal of Parallel and Distributed Computing (JPDC), vol. 69, no. 12, December 2009. Preprint: doi:10.1016/j.jpdc.2009.08.004 .
Li Ou, Xubin He, and Jizhong Han, “An Effective Design for Fast Memory Registration in RDMA”, Journal of Network and Computer Applications, Vol. 32, no. 3, 2009.
Li Ou and Xubin He, “iPVFS: A High Performance Parallel File System over iSCSI for Cluster Computing”,International Journal of Computers and Their Application, Vol. 14, No.4, December 2007, pp. 211-222.
Christian Engelmann, Stephen Scott, C. Box Leangsuksun, and Xubin He, “Symmetric Active/Active High Availability for High-Performance Computing Services”, the Journal of Computers, Vol. 1, No. 8, December 2006, pp. 43-54.
Christian Engelmann, Stephen L. Scott, Chokchai (Box) Leangsuksun, and Xubin He. Symmetric active/active high availability for high-performance computing system services: Accomplishments and limitations. In Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid (CCGrid) 2008: Workshop on Resiliency in High Performance Computing (Resilience) 2008, Lyon, France, May 19-22, 2008. IEEE Computer Society, Los Alamitos, CA, USA.
Christian Engelmann, Stephen Scott, C. Leangsuksun, and Xubin He, “Symmetric Active/Active Replication for Dependent Services,”Proceedings of International Conference on Availability, Reliability and Security (ARES) 2008, Barcelona, Spain, March 4-7, 2008 (Acceptance rate: 21%).
Ben Eckart, Xubin He, Qishi Wu, “Performance Adaptive UDP for High-Speed Bulk Data Transfer over Dedicated Links”, 22nd IEEE International Parallel and Distributed Processing Symposium (IPDPS 2008), April 14-18, 2008 (acceptance rate: 25%).
Li Ou, Christian Engelmann, Xubin He, Xin Chen, and Stephen Scott, “Symmetric Active/Active Metadata Service for Highly Available Cluster Storage Systems,” Proceedings of the 19th International Conference on Parallel and Distributed Computing and Systems (PDCS), Cambridge, Massachusetts, November 19-21, 2007.
Li Ou, Xubin He, Christian Engelmann, and Stephen Scott, “A Fast Delivery Protocol for Total Order Broadcasting,” Proceedings of 16th International Conference on computer Communications and Networks (ICCCN 2007), Honolulu, Hawaii, August 13-16, 2007, acceptance rate: 29% out of 550 submissions.
Christian Engelmann, Stephen Scott, C. Leangsuksun, and Xubin He, “Transparent Symmetric Active/Active Replication for Service-Level High Availability”, Proc. Of the 7th IEEE International Symposium on Cluster Computing and the Grid (CCGRID), May 14-17, 2007. pp. 755-760.
Christian Engelmann, Stephen Scott, C. Leangsuksun, and Xubin He, “On Programming Models for Service-Level High Availability,” Proc. Of the 2nd International conference on Availability, Reliability and Security (ARES), pp. 999-1006, Vienna, Austria, April 2007.
Christian Engelmann, Stephen Scott, Box, and Xubin He, “Towards High Availability for High-Performance Computing System Services: Accomplishments and Limitations”, High Availability and Performance Computing Workshop, Santa Fe, New Mexico, October, 2006.
Li Ou, Xin Chen, Xubin He, Christian Engelmann, and Stephen Scott, “Achieving Computational Efficiency in a High Performance Cluster using Multi-core Processors”, High Availability and Performance Computing Workshop, Santa Fe, New Mexico, October, 2006.
Yung-Chin Fang, Jenwei Hsieh, Victor Mashayekhi, Rooholamini Reza, Stephen Scott, C. Box Leangsuksun, and Xubin He, “Unified Hybrid Computing Infrastructure Management Architecture, High Availability and Performance Computing Workshop, Santa Fe, New Mexico, October, 2006.
Li Ou, Xubin He, and Jizhong Han, "A Fast Read/Write Process to Reduce RDMA Communication Latency," Proceedings of the International Workshop on Networking, Architecture, and Storages (IWNAS), Shenyang, China, August 1-3, 2006.
Li Ou, Xubin He, and Jizhong Han, “MRRC: An effective cache for fast memory registration in RDMA,” Proceedings of the NASA/IEEE Conference on Mass Storage Systems and Technologies (MSST2006), College Park, Maryland, May 2006.
Christian Engelmann, Stephen Scott, C. Leangsuksun, and Xubin He, “Active/Active Replication for Highly Available HPC System Services,” Proceedings of International Symposium on Frontiers in Availability, Reliability and Security (FARES) 2006, Vienna, Austria, April 2006.

Software Release

PA-UDP: A Dynamic Performance-Based Flow Control Method for High-Speed Data Transfer