当前位置：首页 > news >正文

Macos系统上搭建Hadoop详细过程

news 2025/11/6 6:34:11

一、环境检查

1、java环境检查

java -versionopenjdk version "21.0.8" 2025-07-15 LTSOpenJDK Runtime Environment Microsoft-11933201 (build 21.0.8+9-LTS)OpenJDK 64-Bit Server VM Microsoft-11933201 (build 21.0.8+9-LTS, mixed mode, sharing)

2、ssh免密检查

Ahdim@Mac-Mini-Pro hadoop % ssh localhostThe authenticity of host 'localhost (127.0.0.1)' can't be established.ED25519 key fingerprint is SHA256:kqg9GIrbTMPOEAy4wtxzz8aOk6y+U94zc14HaTKfip8.This key is not known by any other names.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesWarning: Permanently added 'localhost' (ED25519) to the list of known hosts.(Ahdim@localhost) Password:Last login: Tue Nov  4 15:40:52 2025如果提示如下信息：Ahdim@Mac-Mini-Pro hadoop % ssh localhostssh: connect to host localhost port 22: Connection refused按照如下步骤打开权限：系统设置->通用->共享->远程登录->打开权限

3、创建目录（安装位置）

以下是我的安装位置，仅供参考，根据你的情况定义自己的安装位置mkdir -p /Users/Ahdim/data/server/hadoop
Ahdim@Mac-Mini-Pro hadoop % pwd                                     
/Users/Ahdim/data/server/hadoop

二、下载 & 解压

cd /Library/Hadoop
curl -O https://archive.apache.org/dist/hadoop/core/hadoop-3.3.6/hadoop-3.3.6.tar.gz
sudo tar -zxf hadoop-3.3.6.tar.gz
sudo mv hadoop-3.3.6 hadoop

三、环境配置

export HADOOP_HOME=/Library/Hadoop/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export JAVA_HOME=$(/usr/libexec/java_home)
source ~/.zshrc

$HADOOP_HOME/etc/hadoop路径：

core-site.xml

<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property><property><name>hadoop.tmp.dir</name><value>/Library/Hadoop/hadoop/tmp</value></property>
</configuration>

hdfs-site.xml

<configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.namenode.http-address</name><value>localhost:9870</value></property>
</configuration>

mapred-site.xml

<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property>
</configuration>

yarn-site.xml

<configuration><property><name>yarn.resourcemanager.hostname</name><value>localhost</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>
</configuration>

四、初始化 & 启动

# 首次格式化
hdfs namenode -format
# 一键启动
start-dfs.sh
start-yarn.sh
# 验证
jps # 应看到 NameNode/DataNode/ResourceManager/NodeManager

五、Web UI

HDFS：http://localhost:9870
YARN：http://localhost:8088

六、运行测试示例

echo "hello hadoop" > words.txt
hdfs dfs -mkdir -p /input
hdfs dfs -put words.txt /input
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /input /output
hdfs dfs -cat /output/part-r-00000

查看全文

http://www.dtcms.com/a/572739.html