当前位置：首页 > news >正文

flinkcdc抽取postgres数据

news 2025/11/13 6:03:25

摘要
postgres是通过复制槽来管理日志的，从而确保主从的数据一致，因此flinkcdc用到postgres的复制槽，授权方面和oracle区别蛮大的。

1.postgres授权flinkcdc读取日志

1.1创建具有流复制权限的用户flinkcdccreate user flinkcdc login replication encrypted password '123456';1.2 给用户flinkcdc赋予数据库连接权限
grant connect on database postgres to flinkcdc;1.3 把当前库所有表查询权限赋给用户flinkcdc
grant select on all tables in schema public to flinkcdc;1.4 发布表(要抽取的表都要进行发布，最好不要用create publication dbz_publication for all tables;)
alter publication dbz_publication add table schema_name.表名;1.5 更改复制标识包含更新和删除之前值（要抽取的表，如果有分区，只需要对分区进行操作，主表可以不操作，所有该表的分区都要哦，后面加分区也要同步操作）
alter table 表名 replica identity full;1.4和1.5可以确保flinkcdc捕捉到添加、删除、修改的数据1.6 要注意的是postgres如果表有分表或分区，flinkcdc只能捕捉到分区的数据，捕捉不到主表或分表的数据，如果分区的名称有规律，可以用正则匹配，匹配到的数据，都写入到目标表，目标表可以是其他数据库的表名（理论上和抽取的主表名称一样，或者加上ods这类的前缀）
1.7 需要用到的查询命令（这些都是常用的命令）
1.7.1 查询哪些表已经发布
select * from pg_publication_tables;1.7.2 查看复制标识（为f标识说明设置成功，能抽取到表变更的日志）
select relreplident from pg_class where relname='表名';1.7.3 查询有哪些复制槽
select * from pg_replication_slots;1.7.4 删除复制槽
select pg_drop_replication_slot('slot_name');  1.7.5 查看复制槽日志积压
select slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_insert_lsn(), restart_lsn)) as delay_size
from pg_replication_slots;

2.java关键代码

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//        env.setParallelism(1);// enable checkpointenv.enableCheckpointing(120000);env.getCheckpointConfig().setCheckpointTimeout(600000L);env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);env.setRestartStrategy(RestartStrategies.fixedDelayRestart(3, Time.milliseconds(10)));String tableList = String.join(",", tableNames.stream().map(c ->    SourceConfig.SCHEMA + "." + c).collect(Collectors.toList()));Properties proper = new Properties();proper.setProperty("snapshot.mode", "never");  // never initial
//        proper.setProperty("debezium.slot.name", slotName);proper.setProperty("slot.drop.on.stop", "true");proper.setProperty("database.serverTimezone", "GMT+8"); //设置时区proper.setProperty("decimal.handling.mode", "STRING");SourceFunction<String> sourceFunction = PostgreSQLSource.<String>builder().hostname(SourceConfig.HOST_NAME).port(SourceConfig.PORT).database(SourceConfig.DATABASE_PRODUCE_SPLIT) // monitor postgresdatabase.schemaList(SourceConfig.SCHEMA) // monitor inventory schema
//                .tableList(tableList).username(SourceConfig.USER_NAME).password(SourceConfig.PASSWORD).decodingPluginName("pgoutput") // pg解码插件.slotName(slotName + "_" + SourceConfig.DATABASE_PRODUCE_SPLIT) // 复制槽名称 不能重复（每个任务对应唯一的复制槽）.deserializer(new JsonDebeziumDeserializationSchema()) // converts SourceRecord to JSON String.debeziumProperties(proper).build();SingleOutputStreamOperator<String> process = cdcSource.process(new ProcessFunction<String, String>() {@Overridepublic void processElement(String row, ProcessFunction<String, String>.Context context, Collector<String> collector) throws Exception {// row就是过来的日志}}).name("test" + "_" + SourceConfig.DATABASE_PRODUCET).setParallelism(15);
env.execute("test" + SourceConfig.DATABASE_PRODUCE);// job任务名称

3.pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.test</groupId><artifactId>test-cdc</artifactId><version>1.0-SNAPSHOT</version><properties><java.version>1.8</java.version><maven.compiler.source>${java.version}</maven.compiler.source><maven.compiler.target>${java.version}</maven.compiler.target><fastjson.vsersion>2.0.52</fastjson.vsersion><druid.version>1.2.15</druid.version><flink.version>1.18.0</flink.version><flinkcdc.vsersion>3.0.1</flinkcdc.vsersion><scala.version>2.12</scala.version><lombok.version>1.18.20</lombok.version><postgresql.version>42.2.12</postgresql.version></properties><dependencies><dependency><groupId>org.postgresql</groupId><artifactId>postgresql</artifactId><version>${postgresql.version}</version></dependency><dependency><groupId>com.ververica</groupId><artifactId>flink-connector-postgres-cdc</artifactId><version>${flinkcdc.vsersion}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-table-api-java-bridge</artifactId><version>${flink.version}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-table-planner_${scala.version}</artifactId><version>${flink.version}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-table-runtime</artifactId><version>${flink.version}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-table-common</artifactId><version>${flink.version}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-java</artifactId><version>${flink.version}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-clients</artifactId><version>${flink.version}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-cep</artifactId><version>${flink.version}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-java</artifactId><version>${flink.version}</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-json</artifactId><version>${flink.version}</version></dependency><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>${fastjson.vsersion}</version></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><version>${lombok.version}</version><scope>provided</scope></dependency><dependency><groupId>com.alibaba</groupId><artifactId>druid-spring-boot-starter</artifactId><version>${druid.version}</version></dependency><dependency><groupId>org.apache.doris</groupId><artifactId>flink-doris-connector-1.18</artifactId><version>24.0.0</version></dependency></dependencies><build><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-assembly-plugin</artifactId><version>3.0.0</version><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs></configuration><executions><execution><id>make-assembly</id><phase>package</phase><goals><goal>single</goal></goals></execution></executions></plugin></plugins></build>
</project>