MySQL-dble分库分表方案
场景:
MySQL 的分库分表是解决业务海量数据存储和高并发访问的核心方案,尤其在单机数据库遇到性能瓶颈时不可或缺。
目前市场上的方案:
1、Sharding-JDBC
2、DBLE
3、MyCat
4、MySQL Fabric
本次介绍的是MySQL8.0 + DBLE 3.22.01.1
架构图
注意:
上面的图中用到了LoadBalance(它只是一个程序连接数据库的方式,不了解的朋友 可以用Ai搜索一下)只限于业务程序用java语言的场景,假设业务程序不是用java开发的,那么此时架构 就要做调整了,需要在dble集群前面加Keepalived + lvs,才能保障dble集群的负载均衡。
部署:
DBLE配置文件
bootstrap.cnf
cluster.cnf #当需要做dble集群时要修改这个 配置文件,如果是单个dble节点,则不需要修改配置文件
db.xml #连接mysql节点的连接信息
user.xml #定义dble用户密码,还有可访问的库
sharding.xml #分库分表文件
注:具体关于配置文件的用法祥见官网说明 https://actiontech.github.io/dble-docs-cn/
1、MySQL部署 步骤略
节点信息
1x2.2x.4x.15:3306 主
1x2.2x.4x.65:3306 从
1x2.2x.4x.212:3306 从
注:本文档中部署mysql只用到一个分片,架构图中是可以为N个分片,可根据自身情况部署对应的分片数量
mysql分库和表名
db_0…db_99 共一百个库
每个库中的表
dp_page_inter
dq_project_page
dp_live_collection
flm_mq_data
flm_standard_data
rbn_collect_log
rbi_cscene_collection
os_category
os_china_map
os_vue_big_screen
web_sql_data_sources
web_sql_export_log
web_sql_state
web_sql_s3_param
每个表的分片字段 : tenant_id
算法:哈希
注:mysql实例事先把分库和表都建好,然后创建用户供dble连接mysql
CREATE USER ‘dblename’@‘%’ IDENTIFIED BY ‘XXX’;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, LOCK TABLES, EXECUTE, REPLICATION CLIENT, CREATE ROUTINE, ALTER ROUTINE ON . TO dblename
@%
;
GRANT XA_RECOVER_ADMIN ON . TO dblename
@%
;
2、DBLE 部署
1)部署jdk1.8.0_202
2)部署zk集群,略
1x2.1x.1x.16x:2181,1x2.1x.1x.2x:2181,1x2.1x.1x.16x:2181
3) DBLE节点和配置文件
1x2.2x.x2.1x3 主实例
bootstrap.cnf
grep -vE '^$|^#' bootstrap.cnf
-agentlib:jdwp=transport=dt_socket,server=y,address=8088,suspend=n
-server
-XX:+AggressiveOpts
-Dfile.encoding=UTF-8
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1984
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.host=127.0.0.1
-Xmx4096M
-Xms2048M
-Xss512k
-XX:MaxDirectMemorySize=1024M
-XX:MetaspaceSize=100M
-XX:-OmitStackTraceInFastThrow
-XX:+PrintHeapAtGC
-XX:+PrintGCDateStamps
-Xloggc:./logs/gc_%WRAPPER_TIME_YYYYMMDDHHIISS%_%p.log
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintTenuringDistribution
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0
-XX:+CMSClassUnloadingEnabled
-XX:LargePageSizeInBytes=128M
-XX:+UseFastAccessorMethods
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
-DhomePath=.
-DinstanceName=ushard-dtwins-dble-prod-01
-DinstanceId=32
-DserverId=server-host-1x2.2x.x2.1x3
-DserverPort=8066
-DmanagerPort=9066
-DmaxCon=0
-DNIOFrontRW=8
-DNIOBackendRW=8
-DfrontWorker=8
-DbackendWorker=8
-DfakeMySQLVersion=8.0.28
-DserverBacklog=2048
-DusePerformanceMode=0
-DuseOuterHa=true
-Dcharset=utf8mb4
-DmaxPacketSize=4194304
-DtxIsolation=2
-Dautocommit=1
-DidleTimeout=600000
-DuseCompression=0
-DcapClientFoundRows=false
-DusingAIO=0
-DuseThreadUsageStat=0
-DuseCostTimeStat=0
-DmaxCostStatSize=100
-DcostSamplePercent=1
-DcheckTableConsistency=0
-DcheckTableConsistencyPeriod=1800000
-DprocessorCheckPeriod=1000
-DsqlExecuteTimeout=3000000
-DbackSocketSoRcvbuf=4194304
-DbackSocketSoSndbuf=1048576
-DbackSocketNoDelay=1
-DfrontSocketSoRcvbuf=1048576
-DfrontSocketSoSndbuf=4194304
-DfrontSocketNoDelay=1
-DotherMemSize=4
-DorderMemSize=4
-DjoinMemSize=4
-DbufferPoolChunkSize=4096
-DbufferPoolPageNumber=409
-DbufferPoolPageSize=2097152
-DmappedFileSize=67108864
-DenableCursor=false
-DmaxHeapTableSize=4096
-DuseSqlStat=1
-DbufferUsagePercent=80
-DclearBigSQLResultSetMapMs=600000
-DsqlRecordCount=10
-DmaxResultSet=524288
-DrecordTxn=0
-DtransactionLogBaseDir=txlogs
-DtransactionLogBaseName=server-tx
-DtransactionRotateSize=16
-DxaSessionCheckPeriod=1000
-DxaLogCleanPeriod=1000
-DxaRecoveryLogBaseDir=xalogs
-DxaRecoveryLogBaseName=xalog
-DxaRetryCount=0
-DviewPersistenceConfBaseDir=viewConf
-DviewPersistenceConfBaseName=viewJson
-DmergeQueueSize=1024
-DorderByQueueSize=1024
-DjoinQueueSize=1024
-DuseJoinStrategy=false
-DnestLoopConnSize=4
-DnestLoopRowsSize=2000
-DenableSlowLog=1
-DslowLogBaseDir=slowlogs
-DslowLogBaseName=slow-query
-DflushSlowLogPeriod=1
-DflushSlowLogSize=1000
-DsqlSlowTime=100
-DslowQueueOverflowPolicy=1
-DmaxCharsPerColumn=65535
-DmaxRowSizeToFile=100000
-DenableBatchLoadData=0
-DenableFlowControl=false
-DflowControlHighLevel=4194304
-DflowControlLowLevel=262144
-DenableGeneralLog=0
-DgeneralLogFile=general/general.log
-DgeneralLogFileSize=16
-DgeneralLogQueueSize=4096
-DenableStatistic=0
-DassociateTablesByEntryByUserTableSize=1024
-DfrontendByBackendByEntryByUserTableSize=1024
-DtableByUserByEntryTableSize=1024
-DstatisticQueueSize=4096
-DsamplingRate=0
-DsqlLogTableSize=1024
-DinSubQueryTransformToJoin=false
-DrwStickyTime=1000
-DjoinStrategyType=-1
-DgroupConcatMaxLen=1024
-DenableSessionActiveRatioStat=1
-DenableConnectionAssociateThread=1
-DxaIdCheckPeriod=300
-DenableSqlDumpLog=0
-DsqlDumpLogBasePath=sqldump
-DsqlDumpLogFileName=sqldump.log
-DsqlDumpLogCompressFilePattern=${date:yyyy-MM}/sqldump-%d{MM-dd}-%i.log.gz
-DsqlDumpLogOnStartupRotate=1
-DsqlDumpLogSizeBasedRotate=52428800
-DsqlDumpLogTimeBasedRotate=1
-DsqlDumpLogDeleteFileAge=90d
-DsqlDumpLogCompressFilePath=*/sqldump-*.log.gz
-DqueryForUpdateMaxRowsSize=20000
-DenableMemoryBufferMonitor=0
-DenableMemoryBufferMonitorRecordPool=1
-DtcpKeepIdle=30
-DtcpKeepInterval=10
-DtcpKeepCount=3
cluster.cnf
grep -vE '^$|^#' cluster.cnf
clusterMode=ucore
clusterEnable=true
sequenceHandlerType=2
needSyncHa=false
clusterIP=1x2.1x.1x.16x,1x2.1x.1x.2x,1x2.1x.1x.16x
showBinlogStatusTimeout=60000
grpcTimeout=10
clusterId=ushard-dtwins-prod-01
sequenceInstanceByZk=true
rootPath=universe/dble-v3
clusterPort=5700
db.xml
cat db.xml
<dble:db xmlns:dble="http://dble.cloud/" version="4.0"><dbGroup rwSplitMode="0" name="dbGroup1" delayThreshold="5"><heartbeat>show slave status</heartbeat><dbInstance name="mydb=1" url="1x2.2x.4x.15:3306" password="xxx" user="dblename" maxCon="3000" minCon="2100" disabled="false" readWeight="2" primary="true"><property name="testOnCreate">false</property></dbInstance><dbInstance name="mydb02" url="1x2.2x.4x.65:3306" password="xxx" user="dblename" maxCon="3000" minCon="2100" disabled="false" readWeight="2" primary="false"><property name="testOnCreate">false</property></dbInstance><dbInstance name="mydb03" url="1x2.2x.4x.212:3306" password="xxx" user="dblename" maxCon="3000" minCon="2100" disabled="false" readWeight="2" primary="false"><property name="testOnCreate">false</property></dbInstance></dbGroup>
</dble:db>
user.xml
<dble:user xmlns:dble="http://dble.cloud/" version="4.0"><managerUser name="manage" password="xxx" maxCon="100"/><shardingUser name="root" password="xxx" maxCon="0" schemas="db" readOnly="false"/><shardingUser name="db_rw" password="xxx" maxCon="0" schemas="db" readOnly="false"/><blacklist name="blacklist1"><property name="selelctAllow">true</property></blacklist>
</dble:user>
sharding.xml
<?xml version="1.0"?>
<dble:sharding xmlns:dble="http://dble.cloud/" version="4.0"><schema name="db"><shardingTable name="dp_page_inter" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="interface_id" function="Hash1000"></shardingTable><shardingTable name="dq_project_page" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="page_id" function="Hash1000"></shardingTable><shardingTable name="dp_live_collection" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="collection_id" function="Hash1000"></shardingTable><shardingTable name="flm_mq_data" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="mq_id" function="Hash1000"></shardingTable><shardingTable name="flm_standard_data" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="data_source_id" function="Hash1000"></shardingTable><shardingTable name="rbn_collect_log" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="collect_id" function="Hash1000"></shardingTable><shardingTable name="rbi_cscene_collection" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="collection_id" function="Hash1000"></shardingTable><shardingTable name="os_category" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="module_id" function="Hash1000"></shardingTable><shardingTable name="os_china_map" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="map_id" function="Hash1000"></shardingTable><shardingTable name="os_vue_big_screen" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="id" function="Hash1000"></shardingTable><shardingTable name="web_sql_data_sources" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="data_source_id" function="Hash1000"></shardingTable><shardingTable name="web_sql_export_log" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="exec_log_id" function="Hash1000"></shardingTable><shardingTable name="web_sql_state" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="statement_id" function="Hash1000"></shardingTable><shardingTable name="web_sql_s3_param" shardingNode="dnp$0-99" shardingColumn="tenant_id" incrementColumn="statement_param_id" function="Hash1000"></shardingTable></schema><shardingNode name="dnp$0-99" dbGroup="dbGroup1" database="db_$1-100"></shardingNode><function name="Hash1000" class="Hash"><property name="partitionCount">1</property><property name="partitionLength">1000</property></function>
</dble:sharding>
1x2.2x.x3.2x1 从实例
配置文件同上一个节点的,此处略
分别启动DBLE:
[root@localhost dble]# pwd
/home/dble
[root@localhost dble]# ./bin/dble start
登陆DBLE
mysql -h 1x2.2x.x2.1x3 -uroot -p’xxx’ -P8066