当前位置：首页 > news >正文

云服务器centos 安装hadoop集群

news 来源：原创 2025/6/27 2:26:07

百度搜索云服务器centos 安装hadoop

创建Hadoop用户
sudo useradd hadoop -m -s /bin/bash
sudo passwd hadoop
123456

下载Hadoop
wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz

解压并移动Hadoop到指定目录
tar -zxvf hadoop-3.2.4.tar.gz
sudo mv hadoop-3.2.4 /usr/local/hadoop

配置环境变量
sudo su - hadoop # 切换到hadoop用户
vi ~/.bash_profile
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source ~/.bash_profile

配置Hadoop配置文件
cd /usr/local/hadoop/etc/hadoop
sudo mkdir -p /usr/local/hadoop/dfs/name /usr/local/hadoop/dfs/data
sudo chown -R hadoop:hadoop /usr/local/hadoop/dfs

# core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://节点:9000</value>
</property>
</configuration>

# yarn-site.xml (Hadoop 3)
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>节点</value>
</property>
</configuration>

# hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/dfs/data</value>
</property>
</configuration>

格式化HDFS并启动Hadoop服务
hdfs namenode -format # 只应在第一次运行，或者在删除namenode数据目录后重新运行
start-dfs.sh # 启动HDFS
start-yarn.sh # 启动YARN

http://节点:50070/
http://节点:8088/

问题一
hadoop is not in the sudoers file. This incident will be reported.
sudo visudo
hadoop ALL=(ALL:ALL) ALL

问题二
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
# 在主节点生成密钥
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
# 将公钥复制到所有节点（包括自己）
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
# 测试SSH连接
ssh localhost

问题三
localhost: ERROR: JAVA_HOME is not set and could not be found.
cd /usr/local/hadoop/etc/hadoop
sudo vim hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.8.0_202

问题四
Unit ssh.service could not be found.
Couldn't resolve host name for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=AppStream&infra=stock [Could not resolve host: mirrorlist.centos.org]

相关文章：