当前位置： 首页 > news >正文

使用seatunnel同步磐维数据库数据

news 来源：原创 2025/5/17 5:38:21

1.下载和安装seatunnel 

登录如下链接，下载seatunnel Apache SeaTunnelhttps://seatunnel.apache.org/download/

本次实验环境下载了2.3.8版本

解压
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"

2.下载连接器插件

Central Repository: org/apache/seatunnelhttps://repo.maven.apache.org/maven2/org/apache/seatunnel/

并将其移动至connectors/目录下

-rw-r--r-- 1 root root 11921607 11月  9 2023 seatunnel-transforms-v2-2.3.8.jar
-rw-r--r-- 1 root root   335483 11月  9 2023 connector-fake-2.3.8.jar
-rw-r--r-- 1 root root   196600 11月  9 2023 connector-console-2.3.8.jar
-rw-r--r-- 1 root root 30996454 5月  15 20:11 connector-cdc-opengauss-2.3.8.jar
-rw-r--r-- 1 root root  1025577 5月  16 14:23 connector-jdbc-2.3.8.jar

3.下载驱动

放到lib/目录下

[root@localhost lib]# ls -lrt
总用量 43236
-rw-r--r-- 1 root root 43187714 11月  9 2023 seatunnel-hadoop3-3.1.4-uber.jar
-rw-r--r-- 1 root root  1082113 5月  16 14:17 postgresql-42.6.2.jar
[root@localhost lib]# 

4.编辑配置文件

如下提供2个样例。

query = "select * from zyzaq.t2_qyq” 不带where条件代表全量同步

query = "select * from zyzaq.t2_qyq WHERE id > 2"，带where条件的支持增量同步。

zyzaq.t2_qyq这里的zyzaq，为schema。

单表样例：

[root@localhost apache-seatunnel-2.3.8]# cat panwei_to_panwei.config
env {parallelism = 4job.mode = "BATCH"
}source{Jdbc {url = "jdbc:postgresql://ip:17700/zyzaq"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"query = "select * from zyzaq.t2_qyq WHERE id > 2"}
}transform {}sink {jdbc {url = "jdbc:postgresql://ip:17700/zyzaq_second"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"generate_sink_sql = truedatabase =zyzaq_second table = "zyzaq.t2_qyq"}
}
[root@localhost apache-seatunnel-2.3.8]#

多表同步样例：

[root@localhost apache-seatunnel-2.3.8]# cat panwei_to_panwei_multi_table.config
env {parallelism = 4job.mode = "BATCH"
}source{Jdbc {url = "jdbc:postgresql://ip:17700/zyzaq"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"result_table_name="accounts_src"query = "select * from zyzaq.accounts"}Jdbc {url = "jdbc:postgresql://ip:17700/zyzaq"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"result_table_name="certificate_src"query = "select * from zyzaq.certificate"}
}transform {Sql {source_table_name = "accounts_src"result_table_name = "accounts_dst"query = "select * from accounts_src"}Sql {source_table_name = "certificate_src"result_table_name = "certificate_dst"query = "select * from certificate_src"}
}sink {jdbc {url = "jdbc:postgresql://ip:17700/zyzaq_second"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"generate_sink_sql = truedatabase =zyzaq_second table = "zyzaq.accounts"source_table_name = "accounts_dst"}jdbc {url = "jdbc:postgresql://ip:17700/zyzaq_second"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"generate_sink_sql = truedatabase =zyzaq_secondtable = "zyzaq.certificate"source_table_name = "certificate_dst"}
}
[root@localhost apache-seatunnel-2.3.8]#

5.执行同步命令

采用多表全量同步方式。需提前创建好目标表结构

./bin/seatunnel.sh --config ./panwei_to_panwei_multi_table.config -m local

6.同步日志，如下截取了部分同步的日志

2025-05-16 15:42:05,426 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-13] - received slot release request, jobID: 975664414164254721, slot: SlotProfile{worker=[localhost]:5801, slotID=1, ownerJobID=975664414164254721, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='efefed6d-6f94-4d30-8a74-af2b5f4c631b'}
2025-05-16 15:42:05,427 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-3] - Job SeaTunnel_Job (975664414164254721), Pipeline: [(1/2)] state process is stop
2025-05-16 15:42:05,427 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job SeaTunnel_Job (975664414164254721), Pipeline: [(1/2)] future complete with state FINISHED
2025-05-16 15:42:05,428 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job SeaTunnel_Job (975664414164254721) turned from state RUNNING to FINISHED.
2025-05-16 15:42:05,428 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job SeaTunnel_Job (975664414164254721) state process is stop
2025-05-16 15:42:05,449 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (975664414164254721) end with state FINISHED
2025-05-16 15:42:05,495 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - 
***********************************************Job Statistic Information
***********************************************
Start Time                : 2025-05-16 15:42:03
End Time                  : 2025-05-16 15:42:05
Total Time(s)             :                   2
Total Read Count          :                 171
Total Write Count         :                 171
Total Failed Count        :                   0
***********************************************2025-05-16 15:42:05,496 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-139022] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2025-05-16 15:42:05,498 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-139022] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:51399, qualifier=null, endpoint=[127.0.0.1]:51399, remoteUuid=2d5752f5-cf61-475d-9603-ddb5820a9167, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2025-05-16 15:42:05,499 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-139022] [5.1] Removed connection to endpoint: [localhost]:5801:2895256a-a4d8-40a5-93ba-70dbb24c5d81, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:51399->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2025-05-16 15:42:05.485, lastWriteTime=2025-05-16 15:42:05.449, closedTime=2025-05-16 15:42:05.497, connected server version=5.1}
2025-05-16 15:42:05,499 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-139022] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2025-05-16 15:42:05,500 INFO  [c.h.c.i.ClientEndpointManager ] [hz.main.event-5] - [localhost]:5801 [seatunnel-139022] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:51399, qualifier=null, endpoint=[127.0.0.1]:51399, remoteUuid=2d5752f5-cf61-475d-9603-ddb5820a9167, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=2d5752f5-cf61-475d-9603-ddb5820a9167, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1747381323177, latest clientAttributes=lastStatisticsCollectionTime=1747381323210,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1747381323168,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=6417530880,os.freePhysicalMemorySize=881057792,os.freeSwapSpaceSize=1448751104,os.maxFileDescriptorCount=262144,os.openFileDescriptorCount=54,os.processCpuTime=7280000000,os.systemLoadAverage=0.84,os.totalPhysicalMemorySize=16474963968,os.totalSwapSpaceSize=3221221376,runtime.availableProcessors=8,runtime.freeMemory=278578192,runtime.maxMemory=477626368,runtime.totalMemory=335020032,runtime.uptime=3225,runtime.usedMemory=56441840, labels=[]}
2025-05-16 15:42:05,505 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-139022] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2025-05-16 15:42:05,506 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2025-05-16 15:42:05,506 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] [localhost]:5801 is SHUTTING_DOWN
2025-05-16 15:42:05,510 INFO  [c.h.i.p.i.MigrationManager    ] [hz.main.cached.thread-2] - [localhost]:5801 [seatunnel-139022] [5.1] Shutdown request of Member [localhost]:5801 - 2895256a-a4d8-40a5-93ba-70dbb24c5d81 this master is handled
2025-05-16 15:42:05,517 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] Shutting down connection manager...
2025-05-16 15:42:05,519 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] Shutting down node engine...
2025-05-16 15:42:05,531 INFO  [.c.c.DefaultClassLoaderService] [main] - close classloader service
2025-05-16 15:42:05,540 INFO  [o.a.s.e.s.EventService        ] [event-forwarder-0] - Event forward thread interrupted
2025-05-16 15:42:08,571 INFO  [c.h.i.i.NodeExtension         ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] Destroying node NodeExtension.
2025-05-16 15:42:08,571 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] Hazelcast Shutdown is completed in 3062 ms.
2025-05-16 15:42:08,572 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] [localhost]:5801 is SHUTDOWN
2025-05-16 15:42:08,572 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2025-05-16 15:42:08,572 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2025-05-16 15:42:13,510 INFO  [s.c.s.s.c.ClientExecuteCommand] [ForkJoinPool.commonPool-worker-1] - run shutdown hook because get close signal
[root@localhost apache-seatunnel-2.3.8]#

相关文章：

  • 文章记单词 | 第82篇（六级）
  • 掌握 LangChain 文档处理核心：Document Loaders 与 Text Splitters 全解析
  • uniapp婚纱预约小程序
  • MySQL 8.0 OCP 1Z0-908 131-140题
  • neo4j框架：java安装教程
  • 云轴科技ZStack官网上线Support AI，智能助手助力高效技术支持
  • 泰迪杯特等奖案例深度解析：基于多级二值化与CNN回归的车牌识别系统设计
  • Vue.js教学第二章：Vue实例创建与核心选项全解析
  • 3D生成新突破：阶跃星辰Step1X-3D开源，可控性大幅提升
  • 框架的源码理解——V3中的ref和reactive
  • 微信小程序点击按钮跳转链接并显示
  • OptiStruct实例：3D实体转子分析
  • 吴恩达 Deep Learning（1-36）ppt逐行理解
  • 汽车装配又又又升级，ethernetip转profinet进阶跃迁指南
  • 使用exceljs将excel文件转化为html预览最佳实践（完整源码）
  • C++ 面试问题与答案 (2025)
  • 基于WISE30sec制作中国1km分辨率土壤属性栅格数据（20种属性/0-200cm深度分层）
  • Rust 学习笔记：关于 String 的练习题
  • Python训练营打卡 Day27
  • 青少年编程与数学 02-019 Rust 编程基础 15课题、错误处理
  • 蒲慕明院士：未来数十年不是AI取代人，而是会用AI的人取代不会用的
  • 网易一季度净利增长三成，丁磊：高度重视海外游戏市场
  • 广西等地旱情缓解，水利部针对甘肃启动干旱防御Ⅳ级响应
  • 丹麦外交大臣拉斯穆森将访华
  • 第十二届警博会在京开幕：12个国家和地区835家企业参展
  • “异常”只停留在医院里，用艺术为“泡泡宝贝”加油