English  |  正體中文  |  简体中文  |  Items with full text/Total items : 63273/95970 (66%)
Visitors : 4900712      Online Users : 315
RC Version 7.0 © Powered By DSPACE, MIT. Enhanced by NTU Library & TKU Library IR team.
Scope Tips:
  • please add "double quotation mark" for query phrases to get precise results
  • please goto advance search for comprehansive author search
  • Adv. Search
    HomeLoginUploadHelpAboutAdminister Goto mobile version
    Please use this identifier to cite or link to this item: https://tkuir.lib.tku.edu.tw/dspace/handle/987654321/105687

    Title: HDFS分散式檔案系統之負載均衡演算法設計
    Other Titles: A load-balancing algorithm for Hadoop Distributed File System
    Authors: 林映辰;Lin, Ying-Chen
    Contributors: 淡江大學資訊工程學系碩士班
    Keywords: Hadoop;分散式檔案系統;負載均衡;雲端運算;Distributed file system;Load Balancing;Cloud Computing
    Date: 2015
    Issue Date: 2016-01-22 15:02:40 (UTC+8)
    Abstract: 隨著網路使用的普及以及儲存設備成本的降低,許多企業開始提供雲端服務。像是由Apache軟體基金會發表的Hadoop Distributed File System (HDFS)、Hadoop MapReduce和HBase,而其中HDFS分散式檔案系統更是廣受歡迎。HDFS為Master-Slave的架構,由單一個NameNode及多個存放著DataNode的Rack組成。其中NameNode為Master負責管理DataNode及使用者,而DataNode則為Slave負責儲存檔案。
    HDFS以分散式的原則儲存檔案,每個檔案皆會被切割成固定大小的Block,並且每個Block預設儲存3份Replica於不同的DataNode上。然而HDFS儲存Block的方式Block Placement並未考慮DataNode的儲存使用率,可能造成儲存負載不均衡,因此需要Balancer解決。Balancer為NameNode的一項工具,由管理員下指令執行,它會反覆的將Block從儲存使用率較高的DataNode搬移到儲存使用率較低的DataNode,直到所有DataNode的儲存使用率都介於平均值加減門檻值之間。但Balancer執行時,需大量搬移Block,會造成網路資源的消耗。並且於過往的研究指出,NameNode為系統中的效能瓶頸,由NameNode負責執行Balancer會更加降低HDFS的效能。
    With the advancement of Internet and increasing data demands, many enterprises are offering cloud services to their customers. Among various cloud computing platforms, the Apache Hadoop project has been widely adopted by many large organizations and enterprises. In the Hadoop ecosystem, Hadoop Distributed File System (HDFS), Hadoop MapReduce, and HBase are open source equivalents of the Google proposed Google File System (GFS), MapReduce framework, and BigTable, respectively. To meet the requirement of horizontal scaling of storage in the big data era, HDFS has received significant attention among researchers. HDFS clusters are in a master-slave architecture: there is a single NameNode and a number of DataNodes in each cluster. NameNode is the master responsible for managing the DataNodes and the client accesses. DataNodes are slaves, and are responsible for storing data.
    As the name suggests, HDFS stores files distributedly. Files are divided into fixed-sized blocks, and in default configuration each block has three replicas stored in three different DataNodes to ensure the fault tolerance capability of HDFS. However, Hadoop''s default strategy of allocating new blocks does not take into account of DataNodes’ utilization, which can lead to load imbalance in HDFS. To cope with the problem, NameNode has a built-in tool called Balancer, which can be executed by the system administrator. Balancer iteratively moves blocks from DataNodes in high utilization to those in low utilization to keep each DataNode’s disk utilization within a configurable range centered at the average utilization. The primary cost of using Balancer to achieve load balance is the bandwidth consumed during the movement of blocks. Besides, the previous research shows that the NameNode is the performance bottleneck of HDFS. That is, frequent execution of Balancer by the NameNode may degrade the performance of HDFS.
    Therefore, in this research we would like to design a new load-balancing algorithm by considering all the situations that may influence the load-balancing state. In the proposed algorithm a new role named BalanceNode is introduced to help in matching heavy-loaded and light-loaded nodes, so those light-loaded nodes can share part of the load from heavy-loaded ones. The simulation results show that our algorithm not only can achieve good load-balancing state in the HDFS, but also with minimized movement cost.
    Appears in Collections:[Graduate Institute & Department of Computer Science and Information Engineering] Thesis

    Files in This Item:

    File Description SizeFormat

    All items in 機構典藏 are protected by copyright, with all rights reserved.

    DSpace Software Copyright © 2002-2004  MIT &  Hewlett-Packard  /   Enhanced by   NTU Library & TKU Library IR teams. Copyright ©   - Feedback