How to Deploy a Hadoop Cluster (Part 4)

🌐 中文

English

This article mainly introduces the relevant extended content in the Hadoop deployment process to facilitate further Hadoop deployment operations.

Introduction

In How to Deploy a Hadoop Cluster (Part 3), I have already completed the cloning of virtual machines. Next, we need to understand some additional extended knowledge to facilitate the subsequent Hadoop cluster deployment operations.

Users and User Permissions

During the Hadoop cluster startup process, because the root user has excessive permissions, Hadoop does not allow us to directly use the root user to start the Hadoop cluster to ensure cluster security. Therefore, we need to use a regular user to start the Hadoop cluster and grant the regular user operational permissions for Hadoop so that the regular user can use the Hadoop cluster.

Basic User Commands

In Linux, we can use the following commands to create a user, set a user password, switch users, view the current user, exit a user, and perform other operations.

1
2
3
4
5
6
7
8
9
10
# Create a regular user (execute under root)
useradd username
# Set the regular user password (execute under root)
passwd username
# Switch user
su username
# View the current user
whoami
# Exit the current user
exit

• Note: username is the user name, which can be modified as needed.

  How to Solve the Problem of Hadoop Cluster Startup Failure Due to Using the Root User

  First, move yourself to the directory where Hadoop is located, then use the following command to change the owner and group operational permissions of Hadoop.

  1
  chown -R username:usernameGroup hadoopdirectory
  

• Note: username is the user name, usernameGroup is the user group name, hadoopdirectory is the name of the directory where Hadoop is located. These can be modified as needed. This will grant the regular user operational permissions for the Hadoop folder, allowing the regular user to use the Hadoop cluster.

  Solving SSH Password-Free Login Between Regular Users

  Different users use different SSH keys. Therefore, we need to send the SSH public key of the newly created regular user to other server users in the Hadoop cluster, so that when other users connect to this regular user via SSH, they can perform password-free login. For specific commands, you can refer to the previous content, changing ssh-copy-id hostname to ssh-copy-id username@hostname.

• Note: username is the user name, hostname is the hostname. These can be modified as needed. This operation is only required between the regular users deploying the cluster.

Using scp and rsync Commands for Transfer

scp and rsync are commonly used file transfer commands in Linux. They can be used to transfer files from one server to another, from a server to the local machine, or from the local machine to another server. The specific commands are as follows (it is recommended to configure SSH password-free login before use):

1
scp -r /home/user/source/ username@hostname:/path/to/destination

• Note: source is the source file path, username is the username on the target server, hostname is the hostname of the target server, /path/to/destination is the destination file path.

  1
  rsync -av /home/user/source/ username@hostname:/path/to/destination
  

• Note: source is the source file path, username is the username on the target server, hostname is the hostname of the target server, /path/to/destination is the destination file path.

How to Disable the Firewall

The firewall during Hadoop cluster usage is generally applied to the entire Hadoop cluster. Therefore, we need to disable the firewall on each server so that the Hadoop cluster can ping and be used normally. The specific commands are as follows:

1
2
systemctl stop firewalld
systemctl disable firewalld

• Note: The above commands need to be executed on each Hadoop cluster server.

FAQ

Permission Issues Persist After Switching User and Group When Using a Regular User

If permission issues persist after switching user and group when using a regular user, you can try using the following command to grant the user operational permissions for the Hadoop folder.

1
chmod -R 777 hadoopdirectory

• Note: hadoopdirectory is the name of the directory where Hadoop is located. It can be modified as needed. 777 grants full permissions to everyone. If you do not want to grant full permissions to everyone, you can modify it as needed.

  Why Does Using scp and rsync Sometimes Copy the Directory Itself?

  The trailing slash in /home/user/source/ for scp and rsync cannot be removed. Otherwise, it will copy the directory itself along with the files inside the directory.

  Does rsync Support Direct File or Directory Copying Between Two Remote Hosts?

  rsync does not support direct file or directory copying between two remote hosts. Using it will result in an error. Therefore, if you need to copy files or directories directly between two remote hosts, you must use the scp command.

  What is the Difference Between rsync and scp?

  scp is an overwrite copy. Even if the target file already exists, it will overwrite the target file. rsync is an incremental copy, meaning if the target file already exists, it will only copy the changed parts of the file and will not overwrite the target file.
  Therefore, if you use scp to copy a file the first time, and later modify the file, using rsync to copy the file will copy the changed parts to the target file without overwriting it, and the speed will also be faster.