hadoop集群DataNode起不来：“DiskChecker$DiskErrorExceptio

最近把线上一个配置在拷贝到线下一台机器后，发现 hadoop datanode起不来，总是报这个异常： 2014-03-11 10:38:44,238 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1337291857-192.168.2.5

最近把线上一个配置在拷贝到线下一台机器后，发现hadoop datanode起不来，总是报这个异常：

2014-03-11 10:38:44,238 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1337291857-192.168.2.50-1394505472069 (storage id DS-1593966629-192.168.2.50-50010-1394505524173) service to search002050.sqa.cm4/192.168.2.50:9000
org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid volume failure config value: 1
 at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.
(FsDatasetImpl.java:183)
 at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance
(FsDatasetFactory.java:34)
 at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance
(FsDatasetFactory.java:30)
 at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:920)
 at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:882)
 at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo
(BPOfferService.java:308)
 at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake
(BPServiceActor.java:218)
 at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:660)
 at java.lang.Thread.run(Thread.java:662)

原因是：
dfs.datanode.failed.volumes.tolerated 这个参数直接拷贝了线上的配置为1，
其含义是：The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown. 即datanode可以忍受的磁盘损坏的个数。

在hadoop集群中，经常会发生磁盘只读或者损坏的情况。datanode在启动时会使用dfs.datanode.data.dir下配置的文件夹（用来存储block），若是有一些不可以用且个数>上面配置的值，DataNode就会启动失败。

在线上环境中fs.datanode.data.dir配置为10块盘，所以dfs.datanode.failed.volumes.tolerated设置为1，是允许有一块盘是坏的。而线下的只有一块盘，这volFailuresTolerated和volsConfigured的值都为1，所以会导致代码里面判断失败。

详见hadoop源码的FsDatasetImpl.java的182行：

 // The number of volumes required for operation is the total number 
 // of volumes minus the number of failed volumes we can tolerate.
 final int volFailuresTolerated =
 conf.getInt(DFSConfigKeys.DFS_DATANODE_FAILED_VOLUMES_TOLERATED_KEY,
 DFSConfigKeys.DFS_DATANODE_FAILED_VOLUMES_TOLERATED_DEFAULT);
 String[] dataDirs = conf.getTrimmedStrings(DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY);
 int volsConfigured = (dataDirs == null) ? 0 : dataDirs.length;
 int volsFailed = volsConfigured - storage.getNumStorageDirs();
 this.validVolsRequired = volsConfigured - volFailuresTolerated;
 if (volFailuresTolerated < 0 || volFailuresTolerated >= volsConfigured) {
 throw new DiskErrorException("Invalid volume failure "
 + " config value: " + volFailuresTolerated);
 }
 if (volsFailed > volFailuresTolerated) {
 throw new DiskErrorException("Too many failed volumes - "
 + "current valid volumes: " + storage.getNumStorageDirs() 
 + ", volumes configured: " + volsConfigured 
 + ", volumes failed: " + volsFailed
 + ", volume failures tolerated: " + volFailuresTolerated);
 }

原文地址：hadoop集群DataNode起不来：“DiskChecker$DiskErrorExceptio, 感谢原作者分享。