当前位置：首页 > news >正文

mit6.5840-lab3-3D-SnapShot-25Summer

news 2025/7/9 12:02:32

前言

这一部分还是比较难的（用时1天半），重点在于你是否理解了SnapShot。
最后在全套测试后时间也是符合规定时间

=== RUN   TestInitialElection3A
Test (3A): initial election (reliable network)...... Passed --  time  3.1s #peers 3 #RPCs    64 #Ops    0
--- PASS: TestInitialElection3A (3.12s)
=== RUN   TestReElection3A
Test (3A): election after network failure (reliable network)...... Passed --  time  6.5s #peers 3 #RPCs   156 #Ops    0
--- PASS: TestReElection3A (6.52s)
=== RUN   TestManyElections3A
Test (3A): multiple elections (reliable network)...... Passed --  time  7.1s #peers 7 #RPCs   513 #Ops    0
--- PASS: TestManyElections3A (7.06s)
=== RUN   TestBasicAgree3B
Test (3B): basic agreement (reliable network)...... Passed --  time  1.4s #peers 3 #RPCs    20 #Ops    0
--- PASS: TestBasicAgree3B (1.38s)
=== RUN   TestRPCBytes3B
Test (3B): RPC byte count (reliable network)...... Passed --  time  2.9s #peers 3 #RPCs    54 #Ops    0
--- PASS: TestRPCBytes3B (2.94s)
=== RUN   TestFollowerFailure3B
Test (3B): test progressive failure of followers (reliable network)...... Passed --  time  5.1s #peers 3 #RPCs   104 #Ops    0
--- PASS: TestFollowerFailure3B (5.10s)
=== RUN   TestLeaderFailure3B
Test (3B): test failure of leaders (reliable network)...... Passed --  time  5.8s #peers 3 #RPCs   184 #Ops    0
--- PASS: TestLeaderFailure3B (5.76s)
=== RUN   TestFailAgree3B
Test (3B): agreement after follower reconnects (reliable network)...... Passed --  time  6.5s #peers 3 #RPCs   116 #Ops    0
--- PASS: TestFailAgree3B (6.50s)
=== RUN   TestFailNoAgree3B
Test (3B): no agreement if too many followers disconnect (reliable network)...... Passed --  time  4.3s #peers 5 #RPCs   172 #Ops    0
--- PASS: TestFailNoAgree3B (4.33s)
=== RUN   TestConcurrentStarts3B
Test (3B): concurrent Start()s (reliable network)...... Passed --  time  0.6s #peers 3 #RPCs    14 #Ops    0
--- PASS: TestConcurrentStarts3B (0.63s)
=== RUN   TestRejoin3B
Test (3B): rejoin of partitioned leader (reliable network)...... Passed --  time  7.1s #peers 3 #RPCs   184 #Ops    0
--- PASS: TestRejoin3B (7.13s)
=== RUN   TestBackup3B
Test (3B): leader backs up quickly over incorrect follower logs (reliable network)...... Passed --  time 26.8s #peers 5 #RPCs  2060 #Ops    0
--- PASS: TestBackup3B (26.82s)
=== RUN   TestCount3B
Test (3B): RPC counts aren't too high (reliable network)...... Passed --  time  2.8s #peers 3 #RPCs    44 #Ops    0
--- PASS: TestCount3B (2.79s)
=== RUN   TestPersist13C
Test (3C): basic persistence (reliable network)...... Passed --  time  6.9s #peers 3 #RPCs   116 #Ops    0
--- PASS: TestPersist13C (6.95s)
=== RUN   TestPersist23C
Test (3C): more persistence (reliable network)...... Passed --  time 18.3s #peers 5 #RPCs   460 #Ops    0
--- PASS: TestPersist23C (18.31s)
=== RUN   TestPersist33C
Test (3C): partitioned leader and one follower crash, leader restarts (reliable network)...... Passed --  time  2.1s #peers 3 #RPCs    40 #Ops    0
--- PASS: TestPersist33C (2.06s)
=== RUN   TestFigure83C
Test (3C): Figure 8 (reliable network)...... Passed --  time 38.2s #peers 5 #RPCs   628 #Ops    0
--- PASS: TestFigure83C (38.20s)
=== RUN   TestUnreliableAgree3C
Test (3C): unreliable agreement (unreliable network)...... Passed --  time  6.5s #peers 5 #RPCs   248 #Ops    0
--- PASS: TestUnreliableAgree3C (6.50s)
=== RUN   TestFigure8Unreliable3C
Test (3C): Figure 8 (unreliable) (unreliable network)...... Passed --  time 47.6s #peers 5 #RPCs  3748 #Ops    0
--- PASS: TestFigure8Unreliable3C (47.64s)
=== RUN   TestReliableChurn3C
Test (3C): churn (reliable network)...... Passed --  time 17.7s #peers 5 #RPCs   668 #Ops    0
--- PASS: TestReliableChurn3C (17.72s)
=== RUN   TestUnreliableChurn3C
Test (3C): unreliable churn (unreliable network)...... Passed --  time 17.3s #peers 5 #RPCs   668 #Ops    0
--- PASS: TestUnreliableChurn3C (17.30s)
=== RUN   TestSnapshotBasic3D
Test (3D): snapshots basic (reliable network)...... Passed --  time  8.1s #peers 3 #RPCs   150 #Ops    0
--- PASS: TestSnapshotBasic3D (8.08s)
=== RUN   TestSnapshotInstall3D
Test (3D): install snapshots (disconnect) (reliable network)...... Passed --  time 65.0s #peers 3 #RPCs  1315 #Ops    0
--- PASS: TestSnapshotInstall3D (65.04s)
=== RUN   TestSnapshotInstallUnreliable3D
Test (3D): install snapshots (disconnect) (unreliable network)...... Passed --  time 83.5s #peers 3 #RPCs  1652 #Ops    0
--- PASS: TestSnapshotInstallUnreliable3D (83.54s)
=== RUN   TestSnapshotInstallCrash3D
Test (3D): install snapshots (crash) (reliable network)...... Passed --  time 79.4s #peers 3 #RPCs  1338 #Ops    0
--- PASS: TestSnapshotInstallCrash3D (79.41s)
=== RUN   TestSnapshotInstallUnCrash3D
Test (3D): install snapshots (crash) (unreliable network)...... Passed --  time 94.5s #peers 3 #RPCs  1586 #Ops    0
--- PASS: TestSnapshotInstallUnCrash3D (94.52s)
=== RUN   TestSnapshotAllCrash3D
Test (3D): crash and restart all servers (unreliable network)...... Passed --  time 22.7s #peers 3 #RPCs   384 #Ops    0
--- PASS: TestSnapshotAllCrash3D (22.72s)
=== RUN   TestSnapshotInit3D
Test (3D): snapshot initialization after crash (unreliable network)...... Passed --  time  6.3s #peers 3 #RPCs   106 #Ops    0
--- PASS: TestSnapshotInit3D (6.25s)
PASS
ok  	6.5840/raft1	595.339sCPU时间: 12.18s
实时时间: 595.65s
最大内存: 228744KB

实验内容

Raft 算法中实现快照（Snapshot）主要出于以下几个重要原因：

减少日志存储空间占用

日志不断增长问题：在 Raft 协议里，领导者会把客户端的请求作为日志条目（Log Entries）复制到所有跟随者节点上。随着系统的持续运行，日志会不断变长。如果没有快照机制，日志会无限制地增长，这会占用大量的磁盘存储空间。
快照压缩日志：快照机制允许将系统的当前状态以快照的形式保存下来，并且丢弃该快照之前的日志条目。这样可以显著减少日志文件的大小，节省磁盘空间。例如，对于一个数据库系统，快照可以记录数据库在某个时间点的完整状态，之后就无需再保留生成该快照之前的操作日志。

加速节点恢复和新节点加入

节点崩溃恢复：当一个节点崩溃后重启时，它需要重新应用日志条目来恢复到崩溃前的状态。如果日志非常长，恢复过程会非常耗时。通过使用快照，节点可以直接加载最新的快照，快速恢复到接近崩溃前的状态，然后只需要应用快照之后的少量日志条目，从而大大缩短恢复时间。
新节点加入集群：当有新节点加入 Raft 集群时，它需要从领导者那里获取完整的日志来同步状态。如果日志过长，传输和应用这些日志会花费大量的时间和网络带宽。而使用快照，新节点可以先下载最新的快照，快速建立起基本状态，再通过复制少量后续日志来完成状态同步，提高新节点加入集群的效率。

降低内存和性能开销

内存使用：在 Raft 节点处理日志时，需要在内存中维护日志的副本。随着日志的增长，内存的使用量也会不断增加，可能会导致内存资源紧张，影响系统的性能。快照机制可以减少需要在内存中维护的日志量，降低内存开销。
日志应用性能：应用大量的日志条目需要消耗大量的 CPU 时间。通过定期创建快照，减少需要应用的日志条目数量，可以提高日志应用的性能，使系统能够更快地响应客户端请求。

本次实验需要完成以下三个部分：

完成SnapShot函数的编写，该函数由上层去调用，调用时已经创建好了一个snapshot，你只需要判断index是否合法（index需要比已应用的下标大，index需要比当前快照的lastIncludedIndex要大），如果合法，则删除掉index之前的日志，并调用persist持久化
完成InstallSnapShot函数的编写，在发送心跳时，我们知道分为两种心跳，一种是带日志的，一种是不带日志的。当Leader发现Follower的nextIndex比自己现在日志的最小下标小时，就发送快照给Follower。此函数为Follower处理snapshot函数。首先应该判断这个snapshot是否合法（同上），则把自己的snapshot替换，并修改日志（将lastIncludedIndex之前的日志删除掉），并且把该snapshot提交给应用机
修改之前所有处理log的地方，因为牵扯到全局索引和局部索引的问题。真正的索引 = lastIncludedIndex + 局部索引。要防止访问越界的问题。可以将lastIncludedIndex保存在log[0]中。真正的索引你可以理解为3A-C中你使用的索引，而3D中对Log进行了裁剪，所以出现了局部索引。

具体实现

func (rf *Raft) Snapshot(index int, snapshot []byte) {// Your code here (3D).rf.mu.Lock()defer rf.mu.Unlock()lastIncludedIndex := rf.GetFirstIndex()// 如果这个快照已存在更新的，就忽略if index <= lastIncludedIndex {return}// 如果这个快照的index比已应用的大，就忽略if index > rf.lastApplied {return}term := rf.log[rf.GetLocalIndex(index)].Term// 把snapshot记录在log[0]rf.log = rf.log[rf.GetLocalIndex(index):]rf.log[0].Command = nilrf.log[0].Term = termrf.log[0].Index = indexraftstate := rf.GetRaftState()rf.persister.Save(raftstate, snapshot)DPrintf("Peers %v ,Snapshot: index=%v, term=%v,leaderLogLen=%v", rf.me, index, term, len(rf.log))
}

// Follower 收到 Leader 的 InstallSnapshot RPC 请求
func (rf *Raft) InstallSnapshot(args *InstallSnapshotArgs, reply *InstallSnapshotReply) {rf.mu.Lock()if args.Term < rf.currentTerm {reply.Term = rf.currentTermrf.mu.Unlock()return}if args.Term > rf.currentTerm {rf.backToFollower(args.Term)}// 重置getHeartbeatrf.resetGetHeartbeat()reply.Term = rf.currentTerm// 如果发来的快照过期了，就忽略if rf.commitIndex >= args.LastIncludedIndex {rf.mu.Unlock()return}// 修改Logif rf.GetLastIndex() > args.LastIncludedIndex {rf.log = rf.log[rf.GetLocalIndex(args.LastIncludedIndex):]} else {rf.log = make([]LogEntity, 1)}rf.log[0].Command = nilrf.log[0].Term = args.LastIncludedTermrf.log[0].Index = args.LastIncludedIndex// 保存快照raftstate := rf.GetRaftState()rf.persister.Save(raftstate, args.Data)// 应用快照rf.commitIndex, rf.lastApplied = args.LastIncludedIndex, args.LastIncludedIndexDPrintf("[FOLLOWER %v] InstallSnapshot: commitIndex=%v, lastApplied=%v, log[0].Index=%v, logLen=%v", rf.me, rf.commitIndex, rf.lastApplied, rf.log[0].Index, len(rf.log))rf.mu.Unlock()// 应用到状态机rf.applyCh <- raftapi.ApplyMsg{CommandValid:  false,Snapshot:      args.Data,SnapshotTerm:  args.LastIncludedTerm,SnapshotIndex: args.LastIncludedIndex,SnapshotValid: true,}
}