Apache SeaTunnel配置使用案例

前置操作

Apache SeaTunnel详解与部署（最新版本2.3.11）-CSDN博客

mkdir /usr/local/soft/apache-seatunnel-2.3.11/job/

一、MySQL to HDFS

官方配置参考：

MySQL | Apache SeaTunnel

Hdfs文件 | Apache SeaTunnel

1、配置确认

将mysql-connector-java-8.0.15.jar放入到/usr/local/soft/apache-seatunnel-2.3.11/lib/文件夹中

2、启动Hadoop

3、MySQL数据库表创建

确认MySQL数据库wjobs表user中有数据，没有则进行创建

4、创建seatunnel配置文件

cd /usr/local/soft/apache-seatunnel-2.3.11/job/
touch mysql2hdfs.conf

内容如下：

env {parallelism = 2job.mode = "BATCH"
}
source {Jdbc {url = "jdbc:mysql://node11:3306/wjobs"driver = "com.mysql.cj.jdbc.Driver"connection_check_timeout_sec = 100user = "root"password = "root123"table_path = "wjobs.user"query = "select  * from wjobs.user"}
}sink {HdfsFile {fs.defaultFS = "hdfs://node11:9000"path = "/seatunnel_data/user"file_format_type = "text"field_delimiter = "\t"row_delimiter = "\n"custom_filename = truefile_name_expression = "${transactionId}_${now}"filename_time_format = "yyyy.MM.dd"sink_columns = ["id","username","password"]is_enable_transaction = true
}
}

或

env {parallelism = 2job.mode = "BATCH"
}
source {Jdbc {url = "jdbc:mysql://node11:3306/wjobs"driver = "com.mysql.cj.jdbc.Driver"connection_check_timeout_sec = 100user = "root"password = "root123"table_path = "wjobs.user"query = "select  * from wjobs.user"}
}sink {HdfsFile {fs.defaultFS = "hdfs://node11:9000"path = "/seatunnel_data/user"file_format_type = "text"field_delimiter = "\t"row_delimiter = "\n"is_enable_transaction = true
}
}

 5、执行seatunnel转换命令

cd /usr/local/soft/apache-seatunnel-2.3.11/
./bin/seatunnel.sh --config ./job/mysql2hdfs.conf -m local

注意：此为单次启动，不用启动seatunnel服务 

二、Hive to MySQL

参考：Hive | Apache SeaTunnel

MySQL | Apache SeaTunnel

1、准备数据

确认hive中表及数据、MySQL对应表已创建

2、创建seatunnel配置文件

cd /usr/local/soft/apache-seatunnel-2.3.11/job/
touch hive2mysql.conf

内容如下：

env {parallelism = 2job.mode = "BATCH"
}
source {Hive {table_name = "uoapdb.ext_startup_logs"metastore_uri = "thrift://node11:9083"}
}sink {jdbc {url = "jdbc:mysql://node11:3306/wjobs?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"user = "root"password = "root123"query = "insert into logs(userId,platform) values(?,?)"}
}

3、 启动Hive

nohup hive --service metastore &
nohup hiveserver2 &

4、执行命令

cd /usr/local/soft/apache-seatunnel-2.3.11/
./bin/seatunnel.sh --config ./job/mysql2hdfs.conf -m local