当前位置： 首页 > wzjs >正文

福州品牌网站建设oemwordpress上传ftp失败

wzjs 2025/9/14 0:22:02
福州品牌网站建设oem,wordpress上传ftp失败,手机怎么制作网址链接,设计网页代码源代码1. 准备传输工具 本方案采用 Apache Seatunnel（简称seatunnel）进行MySQL 到 OceanBase 的数据迁移和同步，出于对方案轻量性的考量，我们采用其内置的Zeta引擎来实现，包括全量同步、离线增量同步，以及CDC方案…

1. 准备传输工具

本方案采用 Apache Seatunnel（简称seatunnel）进行MySQL 到 OceanBase 的数据迁移和同步，出于对方案轻量性的考量，我们采用其内置的Zeta引擎来实现，包括全量同步、离线增量同步，以及CDC方案。

1.1 运行环境准备

自行安装运行环境JAVA，推荐版本为8，但理论上高于8的版本也能正常工作。安装后，请确保已正确配置JAVA_HOME

root:~# java -version
openjdk version "17.0.12" 2024-07-16
OpenJDK Runtime Environment (build 17.0.12+7-Debian-2deb11u1)
OpenJDK 64-Bit Server VM (build 17.0.12+7-Debian-2deb11u1, mixed mode, sharing)java

1.2 下载并安装seatunnel

进入官网下载页面，下载适合版本seatunnel : Apache SeaTunnel

我这里选择最新版本2.3.7

下载
wget https://dlcdn.apache.org/seatunnel/2.3.7/apache-seatunnel-2.3.7-bin.tar.gz
解压
tar -zxvf apache-seatunnel-2.3.7-bin.tar.gz

1.3 连接器插件的下载与安装

seatunnel安装包只包含 主体文件与Zeta引擎，连接不同数据源的插件需要手动下载并配置

1.3.1 自动下载方案

通过配置config/plugin_config 文件来指定 你需要的连接器 默认文件里是全方案的 可以根据你的需要增删一些 我们这里只包含这次演示 需要使用的连接库

connector-cdc-mysql
connector-jdbc
connector-fake
connector-console

然后输入命令

sh bin/install-plugin.sh 2.3.7

则开始自动下载连接器

1.3.2  手动下载方案

进入网站Central Repository: org/apache/seatunnel

找寻自己需要的插件例如：

connector-cdc-mysql-2.3.7.jar
connector-console-2.3.7.jar
connector-fake-2.3.7.jar
connector-jdbc-2.3.7.jar
seatunnel-transforms-v2-2.3.7.jar

将文件手动下载之后 放入connectors

1.3.3  验证连接器安装情况
./bin/seatunnel-connector.sh -lSource
FakeSource MySQL-CDC JdbcSink
Jdbc ConsoleTransform
Copy DynamicCompile FieldMapper Filter FilterRowKind JsonPath LLM Replace Split Sql

由于我们是使用JDBC 使用MySQL的连接方式去操作oceanbase 所以还需要下载一个jdbc-mysql

前往MySQL官网下载jdbc

MySQL :: Download MySQL Connector/J (Archived Versions)

将下载的mysql-connector-j-9.0.0.jar 放到 {seatunnel/lib}中

1.4 seatunnel验证

使用config官方自带的v2批操作验证seatunnel是否正常

./bin/seatunnel.sh --config ./config/v2.batch.config.template -m local参数解释:
seatunnel.sh #seatunnel标准启动脚本
config #选择配置脚本
m #运行方式 这里选择本地运行

当您运行该命令时，您可以在控制台中看到它的输出

2022-12-19 11:01:45,417 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - output rowType: name<STRING>, age<INT>
2022-12-19 11:01:46,489 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=1:  SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: CpiOd, 8520946
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=2: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: eQqTs, 1256802974
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=3: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: UsRgO, 2053193072
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=4: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: jDQJj, 1993016602
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=5: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: rqdKp, 1392682764
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=6: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: wCoWN, 986999925
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=7: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: qomTU, 72775247
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=8: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: jcqXR, 1074529204
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=9: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: AkWIO, 1961723427
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=10: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: hBoib, 929089763
2022-12-19 11:01:46,490 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=11: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: GSvzm, 827085798
2022-12-19 11:01:46,491 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=12: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: NNAYI, 94307133
2022-12-19 11:01:46,491 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=13: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: EexFl, 1823689599
2022-12-19 11:01:46,491 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=14: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: CBXUb, 869582787
2022-12-19 11:01:46,491 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=15: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: Wbxtm, 1469371353
2022-12-19 11:01:46,491 INFO  org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=16: SeaTunnelRow#tableId=-1 SeaTunnelRow#kind=INSERT: mIJDt, 995616438

并且结尾有一个 job总结日志

***********************************************Job Statistic Information
***********************************************
Start Time                : 2024-08-29 22:45:29
End Time                  : 2024-08-29 22:45:33
Total Time(s)             :                   4
Total Read Count          :                  32
Total Write Count         :                  32
Total Failed Count        :                   0
***********************************************

2. 全量同步

2.1 测试表创建

创建两张一模一样的表 表结构如下：
CREATE TABLE `table1` (`id` INT NOT NULL AUTO_INCREMENT,`value1` VARCHAR(255) NOT NULL,`value2` VARCHAR(255) ,`value3` VARCHAR(255) ,`value4` VARCHAR(255) ,`value5` VARCHAR(255) ,`created_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,`updated_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,PRIMARY KEY (`id`),UNIQUE INDEX `idx_value1` (`value1`),INDEX `idx_value2_value3` (`value2`, `value3`),INDEX `idx_value3_value4_value5` (`value3`, `value4`, `value5`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;CREATE TABLE `table2` (`id` INT NOT NULL AUTO_INCREMENT,`value1` VARCHAR(255) NOT NULL,`value2` VARCHAR(255) ,`value3` VARCHAR(255) ,`value4` VARCHAR(255) ,`value5` VARCHAR(255) ,`created_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,`updated_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,PRIMARY KEY (`id`),UNIQUE INDEX `idx_value1` (`value1`),INDEX `idx_value2_value3` (`value2`, `value3`),INDEX `idx_value3_value4_value5` (`value3`, `value4`, `value5`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

我们这边使用navicat创建了各100000条数据

2.2 全量同步配置文件编写

表结构建议手动迁移，自动迁移的表结构会有一些问题，并且不会附加索引

2.2.1 单表全量
env {parallelism = 5job.mode = "BATCH"
}
source {Jdbc {url = "jdbc:mysql://127.0.0.1:3306/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"connection_check_timeout_sec = 100user = "xxx"password = "xxx"query = "select * from seatunnel.table1"}
}
sink {jdbc {url = "jdbc:mysql://127.0.0.1:2883/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"user = "xxx@xxx"password = "xxx"# 自动判断sql语句generate_sink_sql = truedatabase = seatunneltable = table1}
}

结果

  ***********************************************Job Statistic Information***********************************************Start Time                : 2024-08-30 15:05:39End Time                  : 2024-08-30 15:05:47Total Time(s)             :                   8Total Read Count          :              100000Total Write Count         :              100000Total Failed Count        :                   0***********************************************
2.2.2 多表全量抽取
env {parallelism = 5job.mode = "BATCH"
}
source {Jdbc {url = "jdbc:mysql://127.0.0.1:3306/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"connection_check_timeout_sec = 100user = "xxx"password = "xxx"table_list = [{table_path = "seatunnel.table1"},{table_path = "seatunnel.table2"query = "select * from seatunnel.table2 where id > 100"}]}
}
sink {jdbc {url = "jdbc:mysql://127.0.0.1:2883/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"user = "xxx@xxx"password = "xxx"# 自动判断sql语句generate_sink_sql = truedatabase = seatunneltable_list = ["seatunnel.table1","seatunnel.table2"]}
}

结果

  ***********************************************Job Statistic Information***********************************************Start Time                : 2024-08-30 15:10:09End Time                  : 2024-08-30 15:10:20Total Time(s)             :                  10Total Read Count          :              200000Total Write Count         :              200000Total Failed Count        :                   0***********************************************

2.3 增量同步配置文件编写

对于增量同步 简单的方法是在文件编写时 通过query 编写 id或updatetime做增量

env {parallelism = 1job.mode = "BATCH"
}
source {Jdbc {url = "jdbc:mysql://127.0.0.1:3306/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"connection_check_timeout_sec = 100user = "xxx"password = "xxx"query = "SELECT * FROM seatunnel.table1 WHERE updatetime > '2024-01-01' "}
}
sink {jdbc {url = "jdbc:mysql://127.0.0.1:2883/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"user = "xxx@xxx"password = "xxx"generate_sink_sql = truedatabase = seatunneltable = table1 }
}

在输出端的时候会根据主键进行 插入与更新操作 但是这种需要从每次配置配置文件的方案比较繁琐 我更加推荐使用dolphinscheduler配合seatunnel进行操作创建一个工作流

1725004844

从输出端获取最大时间或者id 通过 dolphinscheduler的工作流变量进行传输

env {parallelism = 1job.mode = "BATCH"
}
source {Jdbc {url = "jdbc:mysql://127.0.0.1:3306/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"connection_check_timeout_sec = 100user = "xxx"password = "xxx"query = "SELECT * FROM seatunnel.table1 WHERE updatetime > ${max_id} "}
}
sink {jdbc {url = "jdbc:mysql://127.0.0.1:2883/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"user = "xxx@xxx"password = "xxx"generate_sink_sql = truedatabase = seatunneltable = table1 }
}

多表方案同上

2.4 CDC同步配置文件编写

2.4.1 手动同步表结构

由于seatunnel的oceanbase组件还是有所问题 表结构同步 遇到报错比较麻烦 这一步还是手动同步

2.4.2 检查MySQL  binlog状态

赋予用户所需权限

mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password';mysql> FLUSH PRIVILEGES;

检查binlog日志是否开启

mysql> show variables where variable_name in ('log_bin', 'binlog_format', 'binlog_row_image', 'gtid_mode', 'enforce_gtid_consistency');
+--------------------------+----------------+
| Variable_name            | Value          |
+--------------------------+----------------+
| binlog_format            | ROW            |
| binlog_row_image         | FULL           |
| enforce_gtid_consistency | ON             |
| gtid_mode                | ON             |
| log_bin                  | ON             |
+--------------------------+----------------+
5 rows in set (0.00 sec)

如果不一致 请自行更改mysql.cnf文件

在大型数据库创建一致性快照时 可能会存在读超时 请合理配置

interactive_timeoutwait_timeout

在处理完准备工作之后编写配置文件

env {parallelism = 1job.mode = "STREAMING"checkpoint.interval = 10000
}source {MySQL-CDC {base-url = "jdbc:mysql://127.0.0.1:3306/mysql"username = "xxx"password = "xxx@xxx"table-names = ["seatunnel.table1", "seatunnel.table2"]startup.mode = "initial"}
}sink {jdbc {url = "jdbc:mysql://127.0.0.1:2883/mysql?&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"user = "xxx@xxx"password = "xxx"database = "seatunnel"  # 目标数据库table-names = ["seatunnel.table1", "seatunnel.table2"]generate_sink_sql = true     # 自动生成 SQL}
}

正常启动之后 会进行历史数据迁移 再进行cdc变更

注意:

启动之后会根据配置的表和startup.mode选择的模式 进行不同的操作

startup.mode操作如下:initial 启动时同步历史数据，然后同步增量数据earliest 从最早的偏移量启动latest 从最新偏移量启动specific 从用户提供的特定偏移量启动

如果使用specific 需要添加startup.specific-offset.file binlog文件名startup.specific-offset.pos binlog偏转量


文章转载自：

http://0ihNQNdX.nkpmL.cn
http://rg7JhuYs.nkpmL.cn
http://KrksXIM7.nkpmL.cn
http://D5r6FwsF.nkpmL.cn
http://0q5Nghji.nkpmL.cn
http://bRrErW5Y.nkpmL.cn
http://UoBDzV1g.nkpmL.cn
http://1OzQATbd.nkpmL.cn
http://kxeSf4vF.nkpmL.cn
http://W4rDGJUB.nkpmL.cn
http://2Oe1ZzI6.nkpmL.cn
http://xKoETMRb.nkpmL.cn
http://oZqfV1pg.nkpmL.cn
http://zYf5KdAp.nkpmL.cn
http://tOKLLNj5.nkpmL.cn
http://CtHN7EaK.nkpmL.cn
http://ajnFVOJl.nkpmL.cn
http://5b3CEGDp.nkpmL.cn
http://WsHDRCmx.nkpmL.cn
http://7gXBztfU.nkpmL.cn
http://HrpcT9u3.nkpmL.cn
http://AyNJ3a4n.nkpmL.cn
http://k0Q4sgn7.nkpmL.cn
http://31Unvwoq.nkpmL.cn
http://u170XIrd.nkpmL.cn
http://Kpa36szx.nkpmL.cn
http://Vp9ZAwdu.nkpmL.cn
http://pAVx9PlB.nkpmL.cn
http://uCEiL7OY.nkpmL.cn
http://GgAQwqWG.nkpmL.cn
http://www.dtcms.com/wzjs/723504.html

相关文章：

  • 阿里巴巴网站的功能重庆手机网站制作费用
  • 做淘客网站用备案吗wordpress刷权重插件
  • 北京做网站建设的公司排名wordpress页面和菜单
  • 建筑公司网站设计思路济南网站建设公司川芎网络
  • 嘉兴门户网站php mysql 网站模板
  • 灰色网站网站企业网站开发技术期末试题
  • 建站平台是给谁用的百度推广代理商与总公司的区别
  • 电子商务网站建设与安全网站建设基地
  • 中企动力邮箱手机登录入口湖南企业竞价优化服务
  • 有了域名 怎么做网站查询网网站
  • seo关键词是怎么优化的正规seo关键词排名哪家专业
  • 移动端网站的优点建设工程查询市场价网站
  • 网站内容关键词零售电商平台有哪些
  • 无忧网站建设wordpress 简历 插件
  • 微网站在哪个平台上搭建好 知乎岳阳做网站费用
  • 汕头seo建站如何推广自己
  • 做网站的字体河南郑州建设网站
  • ui设计网站设计与网页制作视频教程网站开发有哪些内容
  • 网站设计语言深圳微信商城网站设计公司
  • 北京做网站哪家好青县网站制作
  • 菠菜建设网站wordpress 获取 图片说明
  • 阿里巴巴做网站分录简述网站的推广策略
  • 网站建设柚子网络科技好看的网站链接
  • 中国建设银行招聘网站首页简介网络营销的概念
  • 购物商城有哪些福州短视频seo方法
  • 创意网站设计模板成都品牌设计网站
  • 200M电信宽带做网站中层管理者培训课程有哪些
  • 南昌网站建设报价单广西做网站公司
  • 网站开发 接单成都91获客营销中心
  • 做网站公司 陕西渭南专业电商网站建设