Flink 1.17.2 集群安装部署

Flink集群的安装

1. 1. 集群规划

Ip	host	Server	Note
192.168.10.101	node01	jobManager、TaskManagerRunner	老大和小弟服务
192.168.10.102	node02	TaskManagerRunner	小弟
192.168.10.103	node03	TaskManagerRunner	小弟

注意：本次使用jdk-1.8.0_331和Centos 7.7,经测试jdk11 、jdk1.8.0_362和Centos 9均可使用。

2. 下载Flink的安装包(https://archive.apache.org/dist/flink/flink-1.17.2)

3. 上传安装包(flink-1.17.2-bin-scala_2.12.tgz)到/home

4. 解压安装包到/usr/local

        [root@node01 ~]# tar -zxvf /home/flink-1.17.2-bin-scala_2.12.tgz -C /usr/local/

5. 配置环境变量

[root@node01 ~]# vi /etc/profile
#文件末尾追加如下内容
#flink env
export FLINK_HOME=/usr/local/flink-1.17.2
export PATH=$PATH:$FLINK_HOME/bin:

6. 配置flink的masters配置文件

        [root@node01 flink-1.17.2]# vi /usr/local/flink-1.17.2/conf/masters

        node01:8081

7. 配置flink的workers配置文件

        [root@node01 flink-1.17.2]# vi /usr/local/flink-1.17.2/conf/workers

        node01

        node02

        node03

9. 配置flink的flink-conf.yaml配置文件

        [root@node01 flink-1.17.2]# vi /usr/local/flink-1.17.2/conf/flink-conf.yaml

        jobmanager.rpc.address: node01

        jobmanager.bind-host: 0.0.0.0

        taskmanager.bind-host: 0.0.0.0

        taskmanager.host: node01

        taskmanager.numberOfTaskSlots: 3

        rest.address: node01

        rest.bind-address: 0.0.0.0

10. 分发flink集群到node02和node03

        [root@node01 flink-1.17.2]# scp -r /usr/local/flink-1.17.2/ node02:/usr/local/

        [root@node01 flink-1.17.2]# scp -r /usr/local/flink-1.17.2/ node03:/usr/local/

11. 在node02和node03上修改flink-conf.yaml文件内容

 #node02修改

        [root@node02 flink-1.17.2]# vi /usr/local/flink-1.17.2/conf/flink-conf.yaml

        taskmanager.host: node02

#node03修改

        [root@node03 flink-1.17.2]# vi /usr/local/flink-1.17.2/conf/flink-conf.yaml

        taskmanager.host: node03

12. 在node02和node03上配置环境变量

#分发

[root@node01 flink-1.17.2]# scp /etc/profile node02:/etc/
[root@node01 flink-1.17.2]# scp /etc/profile node03:/etc/

#在node02上刷新和验证

#在node03上刷新和验证

13. 在node01上启动flink集群

 [root@node01 flink-1.17.2]# start-cluster.sh

 #查看进程服务       

[root@node01 flink-1.17.2]# jps
11271 Jps
11149 TaskManagerRunner
10815 StandaloneSessionClusterEntrypoint
[root@node02 ~]# jps
4821 TaskManagerRunner
4893 Jps
[root@node03 ~]# jps
4903 Jps
4826 TaskManagerRunner

14. 查看flink的Web UI

地址：http://192.168.10.101:8081/#/overview

15. 运行flink自带流式案例测试

#安装nc服务
[root@node01 flink-1.17.2]# yum -y install nc#开启6666端口监听
[root@node01 flink-1.17.2]# nc -lk 6666#在开启一个新的回话窗口提交作业
[root@node01 ~]# flink run /usr/local/flink-1.17.2/examples/streaming/SocketWindowWordCount.jar --hostname node01 --port 6666

#查看web ui

#先在nc监听的端口输入数据

#查看实时的结果

[root@node01 ~]# tail -f /usr/local/flink-1.17.2/log/flink-root-taskexecutor-*-node01.out
nihao : 1
flink : 1
hello : 1
hi : 1
hello : 1
hi : 2
hi : 1
hello : 1

16. 批次作业提交测试

#创建数据集
[root@node01 flink-1.17.2]# vi /home/words.txt
hi hello
flink hi this
shi
dfs
dfs
ni hao
hi hi
hi hi
hello
filn
Flink

#运行flink批次程序

[root@node01 flink-1.17.2]# flink run /usr/local/flink-1.17.2/examples/batch/WordCount.jar --input /home/words.txt --output /home/wc

#查看批次结果

到此为止，Flink 1.17.2 简单的Standalone集群及测试完成！！！