当前位置：首页 > news >正文

本地部署Hive集群

news 来源：原创 2025/5/18 11:56:20

规划

服务	机器
Hive本体	部署在Node1
元数据服务所需的关系型数据库(MYSQL)	部署在Node1

安装MYSQL数据库

# 更新密钥
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022

# 安装Mysql yum库
rpm -Uvh http://repo.mysql.com//mysql57-community-release-el7-7.noarch.rpm

# yum安装MysqL
yum -y install mysql-community-server

# 启动MySQL并跳过授权表，使用--user=mysql选项来指定MySQL以mysql用户身份运行
mysqld --skip-grant-tables --skip-networking --user=mysql &

#  使用mysql登录并重置密码
mysql -uroot
USE mysql;
set global validate_password_policy=LOW;
set global validate_password_length=4;
ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
grant all privileges on *.* to root@"%" identified by 'root' with grant option;
FLUSH PRIVILEGES;

# 重启mysql并重新登录
systemctl stop mysqld
systemctl start mysqld
systemctl enable mysqld

mysql -uroot -p

配置Hadoop

Hive的运行依赖于Hadoop(HDFS、Mapreduce、YARN都依赖)
同时涉及到hDFS文件系统的访问，所以需要配置Hadoop的代理用户
即设置hadoop用户允许代理(模拟)其他用户

配置如下内容在Hadoop的core-site.xml中，并分发到其他节点，且重启HDFS集群

  <property>
    <name>hadoop.proxyuser.hadoop.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hadoop.groups</name>
    <value>*</value>
  </property>

下载解压Hive

hive和mysql驱动安装包位置：

apache-hive-3.1.3-bin.tar.gzhttps://archive.apache.org/dist/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gzmysql-connector-java-5.1.34.jarhttps://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.34/mysql-connector-java-5.1.34.jar

tar -zxf apache-hive-3.1.3-bin.tar.gz -C /export/server/

mv mysql-connector-java-5.1.34.jar  /export/server/apache-hive-3.1.3-bin/lib/

ln -s /export/server/apache-hive-3.1.3-bin hive

配置Hive

在Hive的conf目录内，新建hive-env.sh文件，填入以下环境变量内容：

export HADOOP_HOME=/export/server/hadoop
export HIVE_CONF_DIR=/export/server/hive/conf
export HIVE_AUX_JARS_PATH=/export/server/hive/lib

在Hive的conf目录内，新建hive-site.xml文件，填入以下内容：

vim hive-site.xml

<configuration>
<!-- 存储元数据mysql相关配置 -->
<property>
	<name>javax.jdo.option.ConnectionURL</name>
	<value>jdbc:mysql://node1:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false&amp;useUnicode=true&amp;characterEncoding=UTF-8</value>
</property>

<property>
	<name>javax.jdo.option.ConnectionDriverName</name>
	<value>com.mysql.jdbc.Driver</value>
</property>

<property>
	<name>javax.jdo.option.ConnectionUserName</name>
	<value>root</value>
</property>

<property>
	<name>javax.jdo.option.ConnectionPassword</name>
	<value>hadoop</value>
</property>

<!-- H2S运行绑定host -->
<property>
    <name>hive.server2.thrift.bind.host</name>
    <value>node1</value>
</property>

<!-- 远程模式部署metastore metastore地址 --> # 远程模式，metastore必须单独配置
<property>
    <name>hive.metastore.uris</name>
    <value>thrift://node1:9083</value>
</property>

<!-- 关闭元数据存储授权  --> 
<property>
    <name>hive.metastore.event.db.notification.api.auth</name>
    <value>false</value>
</property>
</configuration>

初始化元数据库

mysql -uroot -p123456
create database hive charset utf8;
exit;

cd /export/server/hive/bin
./schematool -initSchema -dbType mysql -verbos

启动Hive(使用Hadoop用户)

chown -R hadoop:hadoop apache-hive-3.1.3-bin hive
cd /export/server/hive/
mkdir logs
nohup bin/hive --service metastore  >> logs/metastore.log 2>&1 &
cd ../bin
./hive