發(fā)布時(shí)間：2018-01-29 21:15

  本文關(guān)鍵詞： 智能電網(wǎng)云平臺(tái) Hadoop 作業(yè)調(diào)度 云計(jì)算 推測(cè)執(zhí)行任務(wù)　出處：《華北電力大學(xué)》2014年碩士論文　論文類型：學(xué)位論文

【摘要】：隨著我國(guó)智能電網(wǎng)事業(yè)的發(fā)展，全國(guó)電力系統(tǒng)互聯(lián)已成為一個(gè)趨勢(shì)，大量的先進(jìn)的數(shù)據(jù)采集與監(jiān)控設(shè)備、相量測(cè)量單元（PMU）、智能電表等被應(yīng)用，現(xiàn)代電力系統(tǒng)正在演變成一個(gè)集聚大數(shù)據(jù)和信息的計(jì)算系統(tǒng)。針對(duì)智能電網(wǎng)對(duì)海量的數(shù)據(jù)存儲(chǔ)和大規(guī)模并行計(jì)算的迫切需求，鑒于電力系統(tǒng)廣域網(wǎng)的完整性，學(xué)者提出了整合網(wǎng)內(nèi)現(xiàn)有計(jì)算和存儲(chǔ)資源，建立電力私有云的概念。Hadoop是主要由HDFS和MapReduce組成的開源云計(jì)算項(xiàng)目，可以部署在普通個(gè)人計(jì)算機(jī)上，從而組成廉價(jià)的云平臺(tái)。作業(yè)的調(diào)度算法對(duì)云計(jì)算有著至關(guān)重要的作用，它是解決作業(yè)在什么地點(diǎn)、什么時(shí)間執(zhí)行的問題。智能電網(wǎng)云平臺(tái)依托于各級(jí)電網(wǎng)的計(jì)算資源，集群中普遍資源存在著節(jié)點(diǎn)異構(gòu)問題，異構(gòu)節(jié)點(diǎn)的執(zhí)行能力的不同和用戶提交作業(yè)任務(wù)量不同，會(huì)導(dǎo)致比較突出的任務(wù)同步問題。 根據(jù)該情況，本文在hadoop平臺(tái)下，給出了一種基于作業(yè)執(zhí)行時(shí)間預(yù)測(cè)的資源優(yōu)化推測(cè)執(zhí)行算法，該算法通過預(yù)先執(zhí)行作業(yè)一部分任務(wù)，通過這些先行任務(wù)預(yù)測(cè)作業(yè)平均和整體的運(yùn)行時(shí)間，同時(shí)將群集中的節(jié)點(diǎn)以執(zhí)行相同作業(yè)所屬任務(wù)的執(zhí)行時(shí)間為參數(shù)，將節(jié)點(diǎn)分為快節(jié)點(diǎn)和慢節(jié)點(diǎn)，而推測(cè)執(zhí)行的任務(wù)只能發(fā)生在快節(jié)點(diǎn)上，該算法結(jié)合任務(wù)執(zhí)行節(jié)點(diǎn)的性能參數(shù)，判斷該任務(wù)是否進(jìn)行推測(cè)執(zhí)行，當(dāng)推測(cè)執(zhí)行發(fā)生時(shí)會(huì)盡可能以局部執(zhí)行的方式執(zhí)行其后備任務(wù)，推測(cè)任務(wù)發(fā)生之前，，該算法會(huì)檢查群集中其它節(jié)點(diǎn)執(zhí)行該任務(wù)的成本是否低于該節(jié)點(diǎn)（主要以inputsplit的所在節(jié)點(diǎn)與執(zhí)行節(jié)點(diǎn)的距離做參考），如果任務(wù)在其它節(jié)點(diǎn)執(zhí)行成本更低，則算法會(huì)放棄本次推測(cè)執(zhí)行。本文通過實(shí)驗(yàn)比較了該算法和、計(jì)算能力調(diào)度算法、公平調(diào)度算法、基于高優(yōu)先級(jí)滑動(dòng)窗口調(diào)度算法的優(yōu)缺點(diǎn)，通過分別代表內(nèi)存、CPU、網(wǎng)絡(luò)等不同類型資源的云計(jì)算應(yīng)用例程WordCount、CPUActivity、URLGet，進(jìn)行三組，每組六次實(shí)驗(yàn)的測(cè)試，結(jié)果表明該算法在任務(wù)的時(shí)間消耗上，推測(cè)執(zhí)行的發(fā)生率，網(wǎng)絡(luò)資源的占用率上均有明顯的減小，整體上縮短了資源的消耗，并提高了任務(wù)的完成速度。因此在一定程度上適合節(jié)點(diǎn)眾多，拓?fù)浣Y(jié)構(gòu)復(fù)雜，節(jié)點(diǎn)差異大的電力系統(tǒng)私有云的作業(yè)調(diào)度的需求。
[Abstract]:With the development of smart grid in China, the interconnection of national power system has become a trend. A large number of advanced data acquisition and monitoring equipment, phasor measurement unit (PMU), intelligent meter and so on have been applied. Modern power system is evolving into a computing system that gathers big data and information. In view of the urgent demand of smart grid for massive data storage and large-scale parallel computing, considering the integrity of power system wide area network (WAN). Scholars put forward the concept of integrating the existing computing and storage resources in the network and establishing the power private cloud. Hadoop is an open source cloud computing project mainly composed of HDFS and MapReduce. It can be deployed on an ordinary personal computer to form a cheap cloud platform. Job scheduling algorithms play a vital role in cloud computing, which is a solution to where jobs are located. The problem of when to execute. The cloud platform of smart grid depends on the computing resources of all levels of power grid, and the problem of node heterogeneity exists in the common resources in the cluster. The different execution ability of heterogeneous nodes and the number of jobs submitted by users will lead to the problem of task synchronization. According to this situation, this paper presents a resource optimization algorithm based on job execution time prediction based on hadoop platform, which performs a part of the job tasks in advance. At the same time, the nodes in the cluster are divided into fast node and slow node with the execution time of the same job as the parameter. The proposed task can only occur on the fast node, and the algorithm combines the performance parameters of the task execution node to determine whether the task is supposed to be executed. When speculation execution occurs, it performs its backup tasks as locally as possible, prior to the occurrence of the speculated task. The algorithm checks whether the cost of performing this task for other nodes in the cluster is lower than that of the node (mainly referenced by the distance between the node where the inputsplit is located and the node of execution). If the cost of task execution in other nodes is lower, then the algorithm will give up this speculative execution. This paper compares the algorithm with the computational power scheduling algorithm and the fair scheduling algorithm through experiments. Based on the advantages and disadvantages of high priority sliding window scheduling algorithm, cloud computing application routine WordCount, which represents different types of resources, such as memory, network and so on, is adopted. CPUActivityTurlGet3 groups, each group of six experiments, the results show that the algorithm in the task of time consumption, the incidence of execution. The occupancy rate of network resources is obviously reduced, the overall consumption of resources is shortened, and the speed of task completion is improved. Therefore, to some extent, it is suitable for many nodes and complex topology. The demand for job scheduling of private cloud in power system with large node difference.
【學(xué)位授予單位】：華北電力大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2014
【分類號(hào)】：TP393.09;TM73

【相似文獻(xiàn)】

相關(guān)期刊論文 前10條

1 徐一哲;沈瑞寒;;智能電網(wǎng)淺析[J];經(jīng)營(yíng)管理者;2009年15期

2 傅書

本文編號(hào)：1474377