问题描述
我是HDFS系统的新手，遇到一个HDFS问题.
我们有一个HDFS文件系统，其名称节点位于服务器上(该服务器名为0002)，数据节点位于其他两台服务器上(这两个服务器分别名为0004和0005).原始数据来自Flume应用程序，并且带有接收器"(Sink).在Flume中作为HDFS.Flume会将原始数据(txt文件)写入服务器0004和0005上的datanode.
因此，原始数据被复制两次并保存在两台服务器下.该系统可以正常工作一段时间，直到有一天停电.重新启动服务器时，将在名称节点(0002)服务器之前重新启动数据节点服务器(0004和0005).在这种情况下，原始数据仍会保存到0004和0005服务器上，但是namenode(0002)上的元数据信息会丢失.块信息损坏.问题是如何在不丢失原始数据的情况下修复损坏的块?
例如，当我们检查名称节点时
hadoop fsck/wimp/contract-snapshot/year=2020/month=6/day=10/snapshottime=1055/contract-snapshot.1591779548475.csv-文件-块-位置 我们在datanode上找到了文件名，但是该块已损坏.相应的文件名为:
blk_1090579409_16840906 当我们进入数据节点(例如0004)服务器时，我们可以通过以下方式搜索该文件的位置
find ./-name``* blk_1090579409 *''; 我们在HDFS系统的虚拟路径下找到了与csv文件相对应的文件"/wimp/contract-snapshot/year = 2020/month = 6/day = 10/snapshottime = 1055/contract-snapshot.1591779548475.csv".该文件保存在文件夹"./subdir0/subdir235/"下.我们可以将其打开并找到正确的格式.相应的.meta数据为二进制格式(?)，我们无法直接读取.
./subdir0/subdir235/blk_1090579409
问题是，考虑到我们已经找到原始文件(blk_1090579409)，我们如何使用而不丢失这些正确的原始文件来还原损坏的HDFS系统?
经过研究，我找到了一个解决方案，该解决方案可能并不有效，但可以解决.如果有人提供更好的解决方案，请告诉我.
整个想法是从HDFS复制所有文件，按年/日/小时/分钟的顺序将这些文件排列到不同的文件夹中，然后将这些文件夹更新到HDFS上.
我有两个用于存储数据的数据节点(0004和0005).总数据大小为10 TB以上.文件夹结构如下(与问题中的相同，一个在Linux上显示，另一个在Windows上显示):
复制因子设置为2，这意味着(如果没有发生错误)每个数据节点将只有一个且只有一个原始文件副本.因此，我们只扫描一个数据节点上的文件夹/文件(在服务器0004上，大约5 TB以上).根据每个文件的修改日期和时间步长，将文件复制到备份服务器/驱动程序上的新文件夹中.幸运的是，在原始文件中，可以使用时间步长信息，例如 2020-03-02T09:25 .我将时间四舍五入到最接近的五分钟，并且父文件夹是每天的文件夹，新创建的文件夹结构为:
每五分钟扫描一次并将数据从datanode复制到新文件夹中的代码写入Pyspark，大约需要2天(我将代码留在晚上运行)来运行所有操作./p>
然后，我可以每天更新HDFS上的文件夹.在HDFS上，文件夹结构如下:
创建的文件夹与HDFS上的文件夹具有相同的结构，命名约定也相同(在复制步骤中，我将每个复制的文件重命名为与HDFS上的约定匹配的文件).
在最后一步中，我编写了JAVA代码以便对HDFS进行操作.经过一些测试，我能够在HDFS上更新每天的数据.IE.它将删除例如在HDFS上的〜/year = 2019/month = 1/day = 2/文件夹下的数据，然后上载新创建的〜/20190102/〜/year = 2019/month = 1/day = 2/.我每天都做这样的手术.然后损坏的块消失，正确的文件被上载到HDFS上的正确路径.
根据我的研究，还可以通过使用Hadoop上的fsimage文件在停电之前找到损坏的块，但这意味着我可能在停电后损坏HDFS上的块.因此，我决定使用上述方法删除损坏的块，同时仍保留原始文件并在HDFS上对其进行更新.
如果有人有更好或更有效的解决方案，请分享！
I am new to the HDFS system and I come across a HDFS question.
We have a HDFS file system, with the namenode on a server (with this server named as 0002) and datanode on two other servers (with these two severs named as 0004 and 0005 respectively). The original data comes from a Flume application and with the "Sink" in the Flume as HDFS. The Flume will write the original data (txt files) into the datanode on the servers 0004 and 0005.
So, the original data is replicated twice and saved under two servers. The system work well for some time until one day there is a power outage. When restarting the servers, the datanode servers (0004 and 0005) are restarted before the namenode (0002) server. In this case, the original data is still saved onto the 0004 and 0005 server, however the metadata information on the namenode(0002) is lost. The block information become corrupt. The question is how to fix the corrupt blocks without losing the original data?
For example, when we check on the namenode
hadoop fsck /wimp/contract-snapshot/year=2020/month=6/day=10/snapshottime=1055/contract-snapshot.1591779548475.csv -files -blocks -locations
We find the filename on the datanode, but the block is corrupt. The corresponding file name is:
blk_1090579409_16840906
When we go to the datanode (e.g. 0004) server, we can search the location of this file by
find ./ -name "*blk_1090579409*"
We have found the the file corresponding to the csv file under the virtual path of the HDFS system "/wimp/contract-snapshot/year=2020/month=6/day=10/snapshottime=1055/contract-snapshot.1591779548475.csv". The file is saved under folder: "./subdir0/subdir235/" and we can open it and find it is in the correct format. The corresponding .meta data is in binary form(?) and we can not read directly.
./subdir0/subdir235/blk_1090579409
The question is, given that we have found the original file (blk_1090579409), how could we restore the corrupt HDFS system using and without losing these correct original files?
After some research, I find a solution, which may be not efficient but works. If someone comes with a better solution, please let me know.
The whole idea is to copy all the files from the HDFS, arrange these files by year/day/hour/minute into different folders and then update these folders onto HDFS.
I have two datanodes (0004 and 0005) where data is stored. The total data size is of the magnitude of 10+ terabytes. The folder structure is as following (it is the same as in the question, one displayed on linux and the other on Windows):
The replication factor is set as 2, which means (if no mistake happens) that each datanode will have one and only one copy of the original file. Therefore, we just scan the folders/files on one datanode (on server 0004, about 5+ terabytes). Based on the modification date and the timestep in each file, copy the files into a new folder on a backup server/driver. Luckily, in the original files, time step information is available, e.g. 2020-03-02T09:25. I round the time to the nearest five minutes, and parent folder is for each day, with the newly created folder structure as:
The code of scan and copy the files from the datanode into the new folders by each five minutes are written in Pyspark and it takes about 2 days (I leave the code to run in the evening) to run all the operation.
Then I can update the folders on HDFS for each day. On HDFS, the folder structure is as follows:
The created folder is with the same structure as on the HDFS, also the naming convention is the same (in the copy step, I rename each copied files to match the convention on HDFS).
In the final step, I write JAVA code in order to do operations to the HDFS. After some testing, I am able to update the data of each day on HDFS. I.e. It will delete e.g. the data under the folder ~/year=2019/month=1/day=2/ on HDFS and then upload all the folders/files under the newly created folder ~/20190102/ up to ~/year=2019/month=1/day=2/ on HDFS. I do such operation for each day. Then the corrupt blocks disappear, and the right files are uploaded to the correct path on HDFS.
According to my research, it is also possible to find the corrupt blocks before the power outage by using the fsimage file on Hadoop, but this means that I may corrupt the blocks on HDFS after the power outage. Therefore, I decide using the described approach to delete the corrupt blocks while still keeping the original files and update them on HDFS.
If anyone has a better or more efficient solution, please share!
这篇关于修复损坏的HDFS文件而不会丢失数据(datanode中的文件仍然存在)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！