当前位置：首页 > news >正文

使用datax通过HbaseShell封装writer和reader同步hbase数据到hbase_踩坑_细节总结---大数据之DataX工作笔记008

news 来源：原创 2025/5/19 11:37:14

最近在做大数据相关功能,有个需求,使用datax同步hbase到hbase中,其中还是有很多细节值得记录:

首先来看一下datax的源码中,如果你使用phoenix创建的表,那么

你就需要使用对应的hbase带有sql字样的,reader和writer.

然后如果你使用datax-web来进行测试的,那么,他默认使用的是hbase11xreader和对应的writer.

这个时候,是可以读出,使用hbase shell去创建的表的,同时也可以读出phoneix创建的表.

关于适用phoenix去操作hbase常用sql:

1.创建表:
CREATE TABLE IF NOT EXISTS "qqqq_tables" ("id" varchar primary key, "source_id" varchar, "table_name" varchar, "table_desc" varchar, "remark" varchar, "create_time" varchar) COLUMN_ENCODED_BYTES=0;CREATE TABLE IF NOT EXISTS "qqqq_tables" ("id" varchar primary key, "source_id" varchar, "table_name" varchar, "table_desc" varchar, "remark" varchar, "create_time" varchar) COLUMN_ENCODED_BYTES=0;2.显示所有表:
show tables3.查询表:
select * from "qdbd_table_meta"4.命令行连接
./sqlline.py 172.19.126.120,172.19.126.121,172.19.126.122:21815.删除表:
drop table "userinfo";6.插入数据:
0: jdbc:phoenix:172.19.126.120,172.19.126.121> upsert into "userinfo" values('4','xiaohuang','xiaohuang@126.com','13256716111','{name:xiaohuang}');
1 row affected (0.14 seconds)7.查询数据添加条件:对应字段要添加双引号
0: jdbc:phoenix:172.19.126.120,172.19.126.121> select * from "userinfo" where "id" = '3';
+----+--------------+---------------------+-------------+-----------------+
| id |     name     |        email        |   mobile    |    son_json     |
+----+--------------+---------------------+-------------+-----------------+
| 3  | xiaoxingxing | xxingxingou@126.com | 13256716188 | {name:xingxing} |
+----+--------------+---------------------+-------------+-----------------+8.使用upsert更新表的某个字段
0: jdbc:phoenix:172.19.126.120,172.19.126.121> upsert into "userinfo" ("id","name") values ('4','xiaohuangzine');
1 row affected (0.017 seconds)
0: jdbc:phoenix:172.19.126.120,172.19.126.121> select * from "userinfo";
+----+---------------+---------------------+-------------+------------------+
| id |     name      |        email        |   mobile    |     son_json     |
+----+---------------+---------------------+-------------+------------------+
| 3  | xiaoxingxing  | xxingxingou@126.com | 13256716188 | {name:xingxing}  |
| 4  | xiaohuangzine | xiaohuang@126.com   | 13256716111 | {name:xiaohuang} |
+----+---------------+---------------------+-------------+------------------+
2 rows selected (0.035 seconds)9.update更新某个字段无法使用:0: jdbc:phoenix:172.19.126.120,172.19.126.121> update "userinfo" set "name" = 'xiaohuangzi