基于HDFS的華圖在線文庫系統(tǒng)數(shù)據(jù)存儲與管理研究
發(fā)布時間：2018-07-17 19:12
【摘要】：作為用戶共享信息的平臺,文庫系統(tǒng)為用戶帶來了效率和方便,然而,隨著用戶數(shù)據(jù)的增長,使用量的加大,文庫資源的形式和種類也越來越多,成指數(shù)倍增長的海量數(shù)據(jù)資源給存儲系統(tǒng)帶來了難題,如何高效存儲和管理這些數(shù)據(jù)成為急待解決的問題。 云存儲技術(shù)的出現(xiàn),使高效存儲和管理這些海量數(shù)據(jù)成為可能。本文選取了當前非常流行的云平臺Hadoop做為在線文庫系統(tǒng)的存儲平臺,利用Hadoop下的云存儲文件系統(tǒng)HDFS存儲和管理在線文庫系統(tǒng)的文檔文件。HDFS只是為了解決一般性的數(shù)據(jù)存儲和管理難題,簡單將其應用于在線文庫系統(tǒng)不能投入實際的應用,必須作必要的改進。在線文庫系統(tǒng)的文檔資料一般為word、pdf、txt之類的文本文件,這些類型的文件都比較小,90%以上的文檔大小在32KB到20MB之間。HDFS的元數(shù)據(jù)信息是存儲在元數(shù)據(jù)節(jié)點的內(nèi)存中,因此在存儲海量的小文件時,會導致HDFS元數(shù)據(jù)節(jié)點(NameNode)內(nèi)存的過量消耗,進而降低整個HDFS系統(tǒng)的存儲容量,因此本文提出了一種將小文件合并成大文件的存儲優(yōu)化方案,有效地減少了元數(shù)據(jù)節(jié)點的內(nèi)存損耗。另一方面,考慮到合并后存取速度的折損,本文還提出了一種數(shù)據(jù)預取機制,該機制包括兩級緩存,通過這兩級緩存可以大大提升用戶文件讀取的流暢度,緩解云存儲元數(shù)據(jù)管理節(jié)點的壓力。圖22幅,表3個,參考文獻60篇。
[Abstract]:As a platform for users to share information, library system brings users efficiency and convenience. However, with the increase of user data and the increase of usage, the forms and types of library resources become more and more. The massive data resources which increase exponentially bring problems to storage system. How to store and manage these data efficiently becomes an urgent problem to be solved. Cloud storage technology makes it possible to store and manage these massive data efficiently. This paper selects Hadoop, a very popular cloud platform, as the storage platform of online library system. Using the cloud storage file system HDFS under Hadoop to store and manage the document files of the online library system. HDFS is just to solve the general problem of data storage and management, and simply applying it to the online library system can not be put into practical application. Necessary improvements must be made. The document material of the online library system is usually a text file such as word _ _ _. Therefore, when storing large amount of small files, it will lead to excessive consumption of memory in HDFS metadata node (name Node), and then reduce the storage capacity of the whole HDFS system. Therefore, this paper proposes a storage optimization scheme to merge small files into large files. The memory consumption of metadata nodes is reduced effectively. On the other hand, considering the loss of access speed after merging, this paper also proposes a data prefetching mechanism, which includes two levels of cache, which can greatly improve the fluency of user file reading. Ease cloud storage metadata management node pressure. There are 22 figures, 3 tables and 60 references.
【學位授予單位】：中南大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP333
本文編號：2130658
[Abstract]:As a platform for users to share information, library system brings users efficiency and convenience. However, with the increase of user data and the increase of usage, the forms and types of library resources become more and more. The massive data resources which increase exponentially bring problems to storage system. How to store and manage these data efficiently becomes an urgent problem to be solved. Cloud storage technology makes it possible to store and manage these massive data efficiently. This paper selects Hadoop, a very popular cloud platform, as the storage platform of online library system. Using the cloud storage file system HDFS under Hadoop to store and manage the document files of the online library system. HDFS is just to solve the general problem of data storage and management, and simply applying it to the online library system can not be put into practical application. Necessary improvements must be made. The document material of the online library system is usually a text file such as word _ _ _. Therefore, when storing large amount of small files, it will lead to excessive consumption of memory in HDFS metadata node (name Node), and then reduce the storage capacity of the whole HDFS system. Therefore, this paper proposes a storage optimization scheme to merge small files into large files. The memory consumption of metadata nodes is reduced effectively. On the other hand, considering the loss of access speed after merging, this paper also proposes a data prefetching mechanism, which includes two levels of cache, which can greatly improve the fluency of user file reading. Ease cloud storage metadata management node pressure. There are 22 figures, 3 tables and 60 references.
【學位授予單位】：中南大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP333
【參考文獻】
相關期刊論文 前10條
1 孔亞楠;余躍;;云計算下的虛擬存儲研究及應用[J];電腦知識與技術(shù);2010年30期
2 鐘偉彬;周梁月;潘軍彪;文錦軍;;云計算終端的現(xiàn)狀和發(fā)展趨勢[J];電信科學;2010年03期
3 趙吉志;李金;姚萃南;;云計算數(shù)據(jù)中心及標準化發(fā)展[J];信息技術(shù)與標準化;2011年03期
4 吳吉義;傅建慶;平玲娣;謝琪;;一種對等結(jié)構(gòu)的云存儲系統(tǒng)研究[J];電子學報;2011年05期
5 朱偉;;網(wǎng)絡虛擬化典型技術(shù)探討[J];廣東通信技術(shù);2011年01期
6 朝樂門;;云計算環(huán)境下的電子文件遷移模型研究[J];檔案學通訊;2013年01期
7 余慶;;分布式文件系統(tǒng)FastDFS架構(gòu)剖析[J];程序員;2010年11期
8 劉田甜;李超;胡慶成;張桂剛;;云環(huán)境下多副本管理綜述[J];計算機研究與發(fā)展;2011年S3期
9 張敬亮;張軍偉;張建剛;許魯;;藍鯨文件系統(tǒng)中元數(shù)據(jù)與數(shù)據(jù)隔離技術(shù)[J];計算機工程;2010年02期
10 許春聰;黃小猛;吳諾;孫寧偉;楊廣文;;分布式文件系統(tǒng)存儲介質(zhì)評測與分析[J];計算機學報;2010年10期
，本文編號：2130658
本文鏈接：http://www.wukwdryxk.cn/kejilunwen/jisuanjikexuelunwen/2130658.html
最近更新
教材專著
