搭建Flink分布式集群

0. 基础环境：

安装JDK

本次使用 jdk-11.0.26_linux-x64_bin.tar.gz

tar -zxvf jdk-11.0.26_linux-x64_bin.tar.gz -C /usr/local/java/

配置环境变量：

vi /etc/profileJAVA_HOME=/usr/local/java/jdk-11.0.26
CLASSPATH=.:${JAVA_HOME}/lib:$CLASSPATH
PATH=$PATH:${JAVA_HOME}/bin
export JAVA_HOME CLASS_PATH PATH

让环境变量生效：

source /etc/profile 

如果没生效就重启服务器

ssh免密码登录

集群内节点之间免密登录

1. 下载

版本：flink-2.0.0-bin-scala_2.12.tgz
地址： https://www.apache.org/dyn/closer.lua/flink/flink-2.0.0/flink-2.0.0-bin-scala_2.12.tgz

2. 安装

通过虚拟机设置共享文件夹将需要的安装包复制到linux虚拟机中 localhost1。虚拟机的共享盘在 /mnt/hgfs/。 将共享盘安装包复制到 存在目标路径/opt/software/

解压缩

cd /opt/software/
tar -zxvf flink-2.0.0-bin-scala_2.12.tgz -C /usr/local/applications/

3，修改FLINK配置

修改 /conf/config.yaml 文件

at localhost1

jobmanager:bind-host: 0.0.0.0rpc:address: localhost1port: 6123taskmanager:bind-host: 0.0.0.0host: localhost1

at localhost2

jobmanager:bind-host: 0.0.0.0rpc:address: localhost1port: 6123taskmanager:bind-host: 0.0.0.0host: localhost2

at localhost3

jobmanager:bind-host: 0.0.0.0rpc:address: localhost1port: 6123taskmanager:bind-host: 0.0.0.0host: localhost3

修改 /conf/masters文件

localhost1:8081

修改 /conf/workers文件

localhost1
localhost2
localhost3

修改 /conf/zoo.cfg 文件 (可以不改)

server.1=localhost1:2888:3888
server.2=localhost2:2888:3888
server.3=localhost3:2888:3888

4. 将Spark软件分发到集群

先关闭防火墙

systemctl stop firewalldsystemctl disable firewalld

将Flink分发到localhost2 和 localhost3

scp -r flink-2.0.0 root@localhost2:/usr/local/applications/flink-2.0.0
scp -r flink-2.0.0 root@localhost3:/usr/local/applications/flink-2.0.0

5, 启动集群

[root@localhost1 flink-2.0.0]# bin/start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host localhost1.
Starting taskexecutor daemon on host localhost1.
Starting taskexecutor daemon on host localhost2.
Starting taskexecutor daemon on host localhost3.

6, 查看WEB页面

http://localhost1:8081/#/overview