当前位置： 首页 > news >正文

mysql 到 doris 挪移数据

news 来源：原创 2025/5/15 14:36:37

工具datax.....

下载地址：http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz

下载以后解压：tar -xvzf datax.tar.gz

然后，理论上就可以直接使用了。但是，datax本身是python2写的，如果需要python3来执行，肯定会报错。

对于这个问题，可以用下面的方法解决：

DataX的python3使用-CSDN博客

当然，如果是python2来执行，此步骤直接跳过...

另外，datax说是python写的，但是其功能却并不是由python实现的，而是由java实现的。因此运行datax，需要先安装java....

首先，先看下自己系统的版本，再觉得安装哪个版本的java

我的是Ubuntu的虚拟机，x86_64的，因此需要下载：

连接：Java Downloads | Oracle 中国

放到虚拟机上，直接解压：

tar -xvzf  jdk-17.0.9_linux-x64_bin.tar.gz

完事以后，也不用干啥，配下环境变量就好。

vim ~/.bashrc

将下面的内容写入其中就好：（注意，jdk-17.0.9_linux-x64_bin.tar.gz文件解压到哪，JAVA_HOME就绑定啥路径，原则就是 JAVA_HOME/bin 目录就是 jdk-17.0.9_linux-x64_bin.tar.gz解压后的文件的bin目录，确保这个即可）

export JAVA_HOME=/usr/lib/jvm
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

完事以后记得执行：

source ~/.bashrc

然后，就可以看到：

java就算是安装完成了。

接下来，看datax要用的一个配置文件，这个文件由我们自定义，就是具体要干啥，需要给datax说清楚的意思。文件名无所谓，后缀需要是json

大致就是这个文件：

我是要从一mysql数据库将数据导入到Doris数据库。

这个json文件可以这么构建：

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "具体的用户",
                        "password": "具体的password",
                        "column": ["start_time", "cluster", "hostname", "ip", "device", "used_time", "data_type", "end_time"],
                        "connection": [
                            {
                                "jdbcUrl": ["jdbc:mysql://mysql_数据库_ip:3306/db_name"],
                                "table": ["table_name"]
                            }
                        ]
                    }
                },
                "writer": {
                    "name": "mysqlwriter",
                    "parameter": {
                        "writeMode": "insert",
                        "username": "具体的用户",
                        "password": "具体的password",
                        "column": ["start_time", "cluster", "hostname", "ip", "device", "used_time", "data_type", "end_time"],
                        "preSql": [],
                        "postSql": [],
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://doris_数据库_ip:9030/db_name",
                                "table": ["table_name"]
                            }
                        ]
                    }
                }
            }
        ],
        "setting": {
            "speed": {
                "channel": 3
            }
        }
    }
}

然后，就可以执行命令：python3 datax/bin/datax.py job.json

照理说，现在就能执行了，但是，我这里出现错误：

这个错，我最终执行命令：rm -rf datax/plugin/*/._*
解决了....

这里可以参考下：DataX3.0：深入探索与实战应用-CSDN博客 

然后，基本上就可以用了：

不过，最终，我这里的命令执行失败的，因为，我的虚拟机连接不到公司的数据库服务器，比较遗憾。但是，大致，应该是可用的。

后来，实践中，始终报这个错：

好吧，后来的后来.......

这是因为datax里面自带的mysql驱动太老旧了，导致始终无法连接到数据库。需要删除老的mysql驱动，换成新的mysql驱动就可以了。 

参照帖子：DataX同步mysql数据报错 无法连接mysql - 阿伦啊 - 博客园

换成这个就可以了。原始的太老了，是 mysql-connector-java-5.1.34.jar（删掉）

最后。。。。

$ python3 datax/bin/datax.py job.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2025-03-14 19:11:54.552 [main] INFO  VMInfo - VMInfo# operatingSystem class => com.sun.management.internal.OperatingSystemImpl
2025-03-14 19:11:54.558 [main] INFO  Engine - the machine info  => 

	osInfo:	Bisheng 11 11.0.12+9
	jvmInfo:	Linux amd64 4.19.90-2107.6.0.0100.oe1.bclinux.x86_64
	cpu num:	16

	totalPhysicalMemory:	-0.00G
	freePhysicalMemory:	-0.00G
	maxFileDescriptorCount:	-1
	currentOpenFileDescriptorCount:	-1

	GC Names	[G1 Young Generation, G1 Old Generation]

	MEMORY_NAME                    | allocation_size                | init_size                      
	CodeHeap 'profiled nmethods'   | 116.37MB                       | 2.44MB                         
	G1 Old Gen                     | 1,024.00MB                     | 970.00MB                       
	G1 Survivor Space              | -0.00MB                        | 0.00MB                         
	CodeHeap 'non-profiled nmethods' | 116.38MB                       | 2.44MB                         
	Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
	Metaspace                      | -0.00MB                        | 0.00MB                         
	G1 Eden Space                  | -0.00MB                        | 54.00MB                        
	CodeHeap 'non-nmethods'        | 7.25MB                         | 2.44MB                         


2025-03-14 19:11:54.576 [main] INFO  Engine - 
{
	"content":[
		{
			"reader":{
				"name":"mysqlreader",
				"parameter":{
					"column":[
						"start_time",
						"cluster",
						"hostname",
						"ip",
						"device",
						"used_time",
						"data_type",
						"end_time"
					],
					"connection":[
						{
							"jdbcUrl":[
								"jdbc:mysql://xxxxxx:3306/gpu_statistics?useSSL=false"
							],
							"table":[
								"hachi_gpu_day_used_time"
							]
						}
					],
					"password":"****************",
					"username":"root"
				}
			},
			"writer":{
				"name":"mysqlwriter",
				"parameter":{
					"column":[
						"start_time",
						"cluster",
						"hostname",
						"ip",
						"device",
						"used_time",
						"data_type",
						"end_time"
					],
					"connection":[
						{
							"jdbcUrl":"jdbc:mysql://xxxxxxx:9030/smart_center?useSSL=false",
							"table":[
								"gpu_day_used_time_hachi_test"
							]
						}
					],
					"password":"**********",
					"postSql":[],
					"preSql":[],
					"username":"root",
					"writeMode":"insert"
				}
			}
		}
	],
	"setting":{
		"speed":{
			"channel":3
		}
	}
}

2025-03-14 19:11:54.589 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2025-03-14 19:11:54.591 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2025-03-14 19:11:54.591 [main] INFO  JobContainer - DataX jobContainer starts job.
2025-03-14 19:11:54.592 [main] INFO  JobContainer - Set jobId = 0
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loads generally unnecessary.
2025-03-14 19:11:54.983 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://172.22.20.8:3306/gpu_statistics?useSSL=false&yearIsDateType=false&zeroDateTiminyInt1isBit=false&rewriteBatchedStatements=true.
2025-03-14 19:11:55.005 [job-0] INFO  OriginalConfPretreatmentUtil - table:[hachi_gpu_day_used_time] has columns:[id,hostname,ip,device,used_time,start_time,end_time,cluster,dat
2025-03-14 19:11:55.540 [job-0] INFO  OriginalConfPretreatmentUtil - table:[gpu_day_used_time_hachi_test] all columns:[
start_time,cluster,hostname,ip,device,used_time,data_type,end_time
].
2025-03-14 19:11:55.574 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
insert INTO %s (start_time,cluster,hostname,ip,device,used_time,data_type,end_time) VALUES(?,?,?,?,?,?,?,?)
], which jdbcUrl like:[jdbc:mysql://172.22.16.154:9030/smart_center?useSSL=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatem
2025-03-14 19:11:55.575 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2025-03-14 19:11:55.575 [job-0] INFO  JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2025-03-14 19:11:55.575 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2025-03-14 19:11:55.576 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2025-03-14 19:11:55.576 [job-0] INFO  JobContainer - Job set Channel-Number to 3 channels.
2025-03-14 19:11:55.581 [job-0] INFO  JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
2025-03-14 19:11:55.582 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2025-03-14 19:11:55.599 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2025-03-14 19:11:55.603 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2025-03-14 19:11:55.604 [job-0] INFO  JobContainer - Running by standalone Mode.
2025-03-14 19:11:55.609 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2025-03-14 19:11:55.615 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2025-03-14 19:11:55.615 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2025-03-14 19:11:55.646 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2025-03-14 19:11:55.651 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select start_time,cluster,hostname,ip,device,used_time,data_type,end_time fro 
] jdbcUrl:[jdbc:mysql://172.22.20.8:3306/gpu_statistics?useSSL=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2025-03-14 19:12:05.633 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTitReaderTime 0.000s | Percentage 0.00%
2025-03-14 19:12:15.645 [job-0] INFO  StandAloneJobContainerCommunicator - Total 12800 records, 856913 bytes | Speed 83.68KB/s, 1280 records/s | Error 0 records, 0 bytes |  All s |  All Task WaitReaderTime 0.150s | Percentage 0.00%
2025-03-14 19:12:25.646 [job-0] INFO  StandAloneJobContainerCommunicator - Total 25088 records, 1683744 bytes | Speed 80.75KB/s, 1228 records/s | Error 0 records, 0 bytes |  All91s |  All Task WaitReaderTime 0.176s | Percentage 0.00%
2025-03-14 19:12:35.647 [job-0] INFO  StandAloneJobContainerCommunicator - Total 37376 records, 2513454 bytes | Speed 81.03KB/s, 1228 records/s | Error 0 records, 0 bytes |  All62s |  All Task WaitReaderTime 0.216s | Percentage 0.00%
2025-03-14 19:12:45.648 [job-0] INFO  StandAloneJobContainerCommunicator - Total 51712 records, 3466011 bytes | Speed 93.02KB/s, 1433 records/s | Error 0 records, 0 bytes |  All69s |  All Task WaitReaderTime 0.265s | Percentage 0.00%
2025-03-14 19:12:55.650 [job-0] INFO  StandAloneJobContainerCommunicator - Total 66048 records, 4430671 bytes | Speed 94.21KB/s, 1433 records/s | Error 0 records, 0 bytes |  All83s |  All Task WaitReaderTime 0.312s | Percentage 0.00%
2025-03-14 19:13:05.651 [job-0] INFO  StandAloneJobContainerCommunicator - Total 80384 records, 5391333 bytes | Speed 93.81KB/s, 1433 records/s | Error 0 records, 0 bytes |  All52s |  All Task WaitReaderTime 0.358s | Percentage 0.00%
2025-03-14 19:13:15.652 [job-0] INFO  StandAloneJobContainerCommunicator - Total 94720 records, 6345816 bytes | Speed 93.21KB/s, 1433 records/s | Error 0 records, 0 bytes |  All71s |  All Task WaitReaderTime 0.410s | Percentage 0.00%
2025-03-14 19:13:25.653 [job-0] INFO  StandAloneJobContainerCommunicator - Total 109056 records, 7314509 bytes | Speed 94.60KB/s, 1433 records/s | Error 0 records, 0 bytes |  Al705s |  All Task WaitReaderTime 0.444s | Percentage 0.00%
2025-03-14 19:13:35.655 [job-0] INFO  StandAloneJobContainerCommunicator - Total 121344 records, 8131830 bytes | Speed 79.82KB/s, 1228 records/s | Error 0 records, 0 bytes |  Al155s |  All Task WaitReaderTime 0.468s | Percentage 0.00%
2025-03-14 19:13:45.656 [job-0] INFO  StandAloneJobContainerCommunicator - Total 133632 records, 8952834 bytes | Speed 80.18KB/s, 1228 records/s | Error 0 records, 0 bytes |  Al309s |  All Task WaitReaderTime 0.492s | Percentage 0.00%
2025-03-14 19:13:55.657 [job-0] INFO  StandAloneJobContainerCom

 

相关文章：

  • IDEA中链接使用mysql数据库
  • 如何在 React 中实现错误边界？
  • 一场因黄焖鸡引发的技术方案大作战
  • js版本之ES12（2021）、ES13（2022）新特性（九）
  • 封装Axios拦截器实现用户无感刷新AccessToken实践指南
  • 简单创建一个Django项目并配置neo4j数据库
  • Scratch 3.0安装包，支持Win7/10/11、Mac电脑手机平板、少儿便编程的启蒙软件。
  • SQL99 多表查询
  • 成功破解加密机制，研究人员解锁LinuxESXi Akira勒索软件
  • 单片机技术
  • C++复试笔记（三）
  • flutter实践：断点调试踩坑
  • 循环遍历 Java 集合中元素的方法总结
  • 前端开发：混合技术栈的应用
  • 基于异构特征融合与轻量级集成学习的软件漏洞挖掘方案设计与Python实现
  • Spring Boot + InfluxDB 实现高效数据存储与查询
  • 总结 HTTPS 的加密流程
  • markdown 转 word 工具 ‌Pandoc‌
  • 微信小程序wx.request接口报错(errno: 600001, errMsg: “request:fail -2:net::ERR_FAILED“)
  • 从以太网 II 到 VLAN 和 Jumbo Frame：数据帧格式解读
  • 有人倒卖试运营门票？上海乐高乐园：这些票存在无法入园风险
  • 加强战略矿产出口全链条管控将重点开展哪些工作？商务部答问
  • 第十二届警博会在京开幕：12个国家和地区835家企业参展
  • 首映｜奥斯卡最佳国际影片《我仍在此》即将公映
  • 马上评丨火车穿村而过多人被撞身亡，亡羊补牢慢不得
  • 中国创面修复学科发起者之一陆树良教授病逝，享年64岁