Wednesday, November 10, 2010

sample configuration xml files for Hadoop 0.20.x

In Hadoop 0.19.x or earlier, there were only one xml file to modify - hadoop-site.xml.
From Hadoop 0.20.x, there are 3 xml files that you have to configure.
They are (1) core-site.xml (2) mapred-site.xml (3) hdfs-site.xml.
Here are sample xml files that set only the minimal and required settings.


NOTE : they are found in your HADOOP_HOME/conf directory.


1. core-site.xml






hadoop.tmp.dir
/home/hadoop/hadoop-0.20.2/hdfs-tmp
A base for other temporary directories.



fs.default.name
hdfs://203.235.211.195:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.







2. mapred-site.xml






mapred.local.dir
/home/hadoop/hadoop-0.20.2/mapred-tmp
Comma-separated list of paths on the local
filesystem where temporary Map/Reduce data is written.




mapred.job.tracker
203.235.211.195:54311
The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.









3. hdfs-site.xml






hadoop.tmp.dir
/home/hadoop/hadoop-0.20.2/tmp
A base for other temporary directories.



dfs.data.dir
/home/hadoop/hadoop-0.20.2/dfs_blk/${user.name}
Comma separated list of paths on the local filesystem of a
DataNode where it should store its blocks.




dfs.default.name
hdfs://203.235.211.195:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.




dfs.replication
3
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.