How to Deploy a Hadoop Cluster (Part Five)

🌐 中文

English

This article mainly introduces the basic configuration of related configuration files during the Hadoop deployment process.

Introduction

In How to Deploy a Hadoop Cluster (Part Four), I have already discussed some considerations during configuration. Next, we need to configure the configuration files for the subsequent startup and functional implementation of the Hadoop cluster.

Configuring HDFS

HDFS (Hadoop Distributed File System) is one of the core components of the Apache Hadoop ecosystem. It is a distributed file system specifically designed for big data storage and processing.

Opening the Configuration File

1
2
3
cd /path/to/hadoopfile/ # Enter the Hadoop installation directory
cd etc/hadoop/ # Enter the Hadoop configuration file directory
cd hdfs-site.xml # Enter the HDFS configuration file

• Note: /path/to/hadoopfile/ is the Hadoop installation directory. Modify it as needed.

  Configuration File Content

1
2
3
4
5
6
7
8
9
10
<configuration>
<property>
  <name>dfs.namenode.http-address</name>
  <value>hadoopname01:9870</value>
</property>
<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>hadoopname02:9868</value>
</property>
</configuration>

• Note: hadoopname01 is the hostname of the namenode, 9870 is the port number of the namenode, hadoopname02 is the hostname of the secondary namenode, and 9868 is the port number of the secondary namenode. Modify them as needed. It is recommended to use different hostnames for hadoopname01 and hadoopname02 to achieve high availability of the Hadoop cluster.

  Configuration Content Explanation

  1. dfs.namenode.http-address - hadoopname01:9870 This property specifies the HTTP address of the HDFS NameNode, used to access the NameNode via the Web UI interface. hadoopname01:port01 represents the host address and port number of the NameNode. Through this address, users can access the HDFS Web UI (typically the NameNode management page) to manage the file system, view the file directory structure, check file block distribution, and perform other operations.
  2. dfs.namenode.secondary.http-address - hadoopname02:9868 This property specifies the HTTP address of the HDFS Secondary NameNode, used to access the Secondary NameNode via the Web UI interface. hadoopname02:6868 represents the host address and port number of the Secondary NameNode. The main function of the Secondary NameNode is to periodically download HDFS metadata (fsimage and edits logs), merge and compress them, and upload the results back to the NameNode. Through this address, users can access the Secondary NameNode’s Web management interface to view its running status, etc.

Configuring YARN

YARN (Yet Another Resource Negotiator) is another core component of the Apache Hadoop ecosystem. It is a resource management and scheduling system used to coordinate and manage computing resources in a cluster.

Opening the Configuration File

1
2
3
cd /path/to/hadoopfile/ # Enter the Hadoop installation directory
cd etc/hadoop/ # Enter the Hadoop configuration file directory
cd yarn-site.xml # Enter the YARN configuration file

• Note: /path/to/hadoopfile/ is the Hadoop installation directory. Modify it as needed.

  Configuration File Content

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoopname03</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

  <property>
    <name>yarn.log.server.url</name>
    <value>http://hadoopname01:19888/jobhistory/logs</value>
  </property>

  <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
  </property>
</configuration>

• Note: hadoopname03 is the hostname of the resourcemanager. Modify it as needed. hadoopname02 is the hostname of the logserver. Modify it as needed.

  Configuration Content Explanation

  1. yarn.nodemanager.aux-services - mapreduce_shuffle This property defines the auxiliary services enabled on the YARN NodeManager. Here, mapreduce_shuffle specifies that the Shuffle process for MapReduce jobs will be handled by the NodeManager. Shuffle is a critical step in MapReduce jobs, responsible for transferring data between the Map and Reduce phases. Enabling the mapreduce_shuffle service means the NodeManager will provide this functionality for MapReduce jobs.
  2. yarn.resourcemanager.hostname - hadoopname03 This property specifies the hostname or address of the YARN ResourceManager. hadoopname03 is the hostname of the YARN resource manager, responsible for managing computing resources in the cluster, scheduling jobs, and coordinating task execution.
  3. yarn.nodemanager.env-whitelist - JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME This property sets the whitelist of environment variables for the NodeManager. When the NodeManager starts, it reads and allows these environment variables from the whitelist. These variables typically point to environment configurations related to Hadoop and Java, such as JAVA_HOME, HADOOP_COMMON_HOME, etc. This setting ensures the NodeManager can access the correct environment variables to properly run and manage tasks.
  4. yarn.log-aggregation-enable - true This property enables YARN’s log aggregation feature. When set to true, YARN aggregates log files from all executing tasks into a centralized location instead of keeping them on individual nodes. This helps simplify log management, especially when dealing with large-scale clusters.
  5. yarn.log.server.url - http://hadoopname01:19888/jobhistory/logs This property specifies the URL address of the log server. http://hadoopname01:19888/jobhistory/logs is the Web interface address of the YARN log server, where logs of jobs in the cluster can be viewed. Users can access this address to view job execution logs, error messages, and detailed running status.
  6. yarn.log-aggregation.retain-seconds - 604800 This property sets the duration (in seconds) for retaining aggregated logs. 604800 seconds is equivalent to 7 days. This means YARN will retain aggregated job logs for 7 days, and logs older than this will be automatically deleted. This setting helps administrators manage storage space and prevent log files from occupying disk space indefinitely.

Configuring MapReduce

MapReduce is the core computational model of the Apache Hadoop ecosystem. It is a programming model and implementation framework for processing large-scale datasets. Inspired by functional programming, MapReduce decomposes complex distributed computing problems into two main phases: Map and Reduce.

Opening the Configuration File

1
2
3
cd /path/to/hadoopfile/ # Enter the Hadoop installation directory
cd etc/hadoop/ # Enter the Hadoop configuration file directory
cd mapred-site.xml # Enter the MapReduce configuration file

• Note: /path/to/hadoopfile/ is the Hadoop installation directory. Modify it as needed.

  Configuration File Content

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<configuration>
<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property>
  <name>mapreduce.jobhistory.address</name>
  <value>hadoopname01:10020</value>
</property>
<property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>hadoopname01:19888</value>
</property>
</configuration>

• Note: hadoopname01 is the hostname of the jobhistory. Modify it as needed. hadoopname01 is the hostname of the logserver. Modify it as needed.

  Configuration Content Explanation

  1. mapreduce.framework.name - yarn This property specifies the execution framework for MapReduce jobs. Setting it to yarn means MapReduce jobs will be scheduled and executed by the YARN (Yet Another Resource Negotiator) framework. YARN is Hadoop’s resource manager, used to dynamically allocate computing resources to different jobs.
  2. mapreduce.jobhistory.address - hadoopname01:10020 This property specifies the address of the MapReduce job history server. hadoopname01:10020 is the server address and port where job history information is saved. After a MapReduce job completes, related job history information is recorded on this server for subsequent querying and analysis.
  3. mapreduce.jobhistory.webapp.address - hadoopname01:19888 This configuration defines the access address for the Web interface of the MapReduce job history server. hadoopname01:19888 is the URL address of the job history Web application. Users can access this address via a browser to view and manage historical data of completed MapReduce jobs, such as job status, execution time, task statistics, and other information.

Configuring core-site

core-site.xml is a core configuration file in the Apache Hadoop ecosystem, containing global configuration information for the Hadoop cluster. This file defines Hadoop’s core properties, such as the Hadoop name, data storage paths, file system implementation classes, etc.

Opening the Configuration File

1
2
3
cd /path/to/hadoopfile/ # Enter the Hadoop installation directory
cd etc/hadoop/ # Enter the Hadoop configuration file directory
cd core-site.xml # Enter the core configuration file

• Note: /path/to/hadoopfile/ is the Hadoop installation directory. Modify it as needed.

  Configuration File Content

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://hadoopname01:8020</value>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/path/to/data</value>
</property>
<property>
  <name>hadoop.http.staticuser.user</name>
  <value>byolio</value>
</property>
</configuration>

• Note: hadoopname01 is the hostname of the namenode. Modify it as needed. /path/to/data is the data storage path. Modify it as needed. byolio is the Hadoop username. Modify it as needed.

  Configuration Content Explanation

  1. fs.defaultFS - hdfs://hadoopname01:8020 This is the internal URI of the default file system for the Hadoop Distributed File System (HDFS). It specifies the NameNode server address and port (hadoopname01:8020). All file operations will default to using this address if a file system is not explicitly specified.
  2. hadoop.tmp.dir - /path/to/data This property sets the path for Hadoop’s temporary directory. Hadoop stores various temporary files and intermediate data in this directory. This includes local storage locations for NameNode and DataNode, temporary files for MapReduce jobs, etc. It is recommended to set this directory on a disk with sufficient space.
  3. hadoop.http.staticuser.user - byolio This configuration defines the static username for Hadoop Web UIs (such as HDFS NameNode Web UI, ResourceManager Web UI, etc.). When accessing HDFS files via the Web interface, this user identity will be used. This is important for proxy access and permission control.

FAQ

Why Configure the History Server Functionality

After a MapReduce job finishes running, the ResourceManager no longer saves detailed information and logs of these jobs. Configuring the history server is necessary to preserve this information, allowing administrators and developers to view execution details of completed jobs for troubleshooting errors.

Why Configure the Log Aggregation Functionality

Without log aggregation enabled, logs on each node are stored locally, making querying, accessing, and analyzing logs complex, especially when the cluster is large. After enabling log aggregation, logs from all jobs are stored centrally in one location, simplifying log search and management.

Why Can’t I Perform Operations Like Deletion on the Hadoop Web UI

If a static username is not configured, the Hadoop Web UI will use the default dr.who user, which does not have permission to perform operations like deletion on the Hadoop Web UI. Therefore, it is necessary to configure the static username to a user who has permission to perform such operations, enabling administrators and developers to operate on the Hadoop Web UI.

How to Find Default Configuration Files and Selectively Configure Them

Read the official documentation and review the source code to find:

1. core-default.xml: Contains default core configuration information for Hadoop, such as the Hadoop name, data storage paths, file system implementation classes, etc.
2. hdfs-default.xml: Contains default configuration information for the Hadoop Distributed File System (HDFS), such as the NameNode server address and port, DataNode local storage locations, MapReduce job temporary files, etc.
3. yarn-default.xml: Contains default configuration information for Hadoop’s resource management and scheduling system (YARN), such as the ResourceManager address and port, MapReduce job execution mode, log aggregation functionality, etc.
4. mapred-default.xml: Contains default configuration information for Hadoop’s MapReduce framework, such as MapReduce job execution mode, log aggregation functionality, history server functionality, etc. Then, based on these default configuration files, selectively configure the information in core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml to achieve a customized Hadoop cluster configuration.

What Are the Commonly Used Configuration Ports

Distributed systems involve a large number of services and processes. If ports are not predefined, different services or processes might use the same port, leading to port conflicts, service unavailability, or system instability. Therefore, everyone uses some common default ports for Hadoop 3.x: NameNode internal communication port: 8020/9000/9820 Hadoop Web UI communication port: 9870 MapReduce Web UI port: 8088 History server communication port: 19888

Summary

This article mainly introduced the basic configuration of Hadoop cluster configuration files, including the configuration of hdfs, yarn, mapreduce, core, and other configuration files. It provided the configuration content and explanations to help you better understand Hadoop cluster configuration files, enabling you to better configure the Hadoop cluster to achieve high availability and high performance.