今天做HA切换测试时,发现Zenoss数据只记录到很早以前的情况。判断是Drbd脑裂。主上显示:

点击(此处)折叠或打开

  1. # service drbd status
  2. drbd driver loaded OK; device status:
  3. version: 8.2.6 (api:88/proto:86-88)
  4. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
  5. m:res cs st ds p mounted fstype
  6. 1:mysql StandAlone Primary/Unknown UpToDate/DUnknown - /var/lib/mysql ext3
  7. 2:zenhome StandAlone Primary/Unknown UpToDate/DUnknown - /opt/zenoss ext3
  8. 3:zenperf StandAlone Primary/Unknown UpToDate/DUnknown - /opt/zenoss/perf ext3
从上显示:

点击(此处)折叠或打开

  1. # service drbd status
  2. drbd driver loaded OK; device status:
  3. version: 8.2.6 (api:88/proto:86-88)
  4. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
  5. m:res cs st ds p mounted fstype
  6. 1:mysql StandAlone Secondary/Unknown UpToDate/DUnknown -
  7. 2:zenhome StandAlone Secondary/Unknown UpToDate/DUnknown -
  8. 3:zenperf StandAlone Secondary/Unknown UpToDate/DUnknown -
对于在从上入手,修复脑裂

点击(此处)折叠或打开

  1. # drbdadm disconnect mysql
  2. # drbdadm disconnect zenhome
  3. # drbdadm disconnect zenperf
  4. # drbdadm secondary mysql
  5. # drbdadm secondary zenhome
  6. # drbdadm secondary zenperf
  7. # drbdadm -- --discard-my-data connect mysql
  8. # drbdadm -- --discard-my-data connect zenhome
  9. # drbdadm -- --discard-my-data connect zenperf
查看从的Drbd状态

点击(此处)折叠或打开

  1. more /proc/drbd
  2. version: 8.2.6 (api:88/proto:86-88)
  3. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-bu
  4. ild, 2008-10-03 11:30:17
  5. 1: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
  6. ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:7584
  7. 2: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
  8. ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:47888
  9. 3: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
  10. ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:1936
显示WFConnection状态。看来,还得重新在主上进行连接。

点击(此处)折叠或打开

  1. # drbdadm connect mysql
  2. # drbdadm connect zenhome
  3. # drbdadm connect zenperf
再次查看主从状态,DRBD脑裂恢复。

点击(此处)折叠或打开

  1. more /proc/drbd
  2. version: 8.2.6 (api:88/proto:86-88)
  3. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-bu
  4. ild, 2008-10-03 11:30:17
  5. 1: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
  6. ns:89456 nr:0 dw:45201920 dr:162898 al:97 bm:98 lo:1 pe:9 ua:253 ap:1 oos:16
  7. 6712
  8. [======>.............] sync'ed: 36.6% (166712/255948)K
  9. finish: 0:00:36 speed: 4,464 (6,864) K/sec
  10. 2: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
  11. ns:47360 nr:0 dw:4351072 dr:532070 al:229 bm:394 lo:1 pe:9 ua:1325 ap:2 oos:
  12. 339072
  13. [=>..................] sync'ed: 13.7% (339072/386240)K
  14. finish: 0:01:04 speed: 5,240 (5,240) K/sec
  15. 3: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
  16. ns:15372 nr:0 dw:8790796 dr:41474 al:66 bm:89 lo:2 pe:6 ua:344 ap:1 oos:1214
  17. 48
  18. [=>..................] sync'ed: 14.8% (121448/136668)K
  19. finish: 0:00:31 speed: 3,804 (3,804) K/sec

点击(此处)折叠或打开

  1. # more /proc/drbd
  2. version: 8.2.6 (api:88/proto:86-88)
  3. GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
  4. 1: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
  5. ns:0 nr:192920 dw:192920 dr:0 al:0 bm:14 lo:1 pe:1980 ua:0 ap:0 oos:63216
  6. [==============>.....] sync'ed: 76.2% (63216/255948)K
  7. finish: 0:00:26 speed: 2,368 (4,816) K/sec
  8. 2: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
  9. ns:0 nr:150944 dw:150944 dr:0 al:0 bm:18 lo:2 pe:7320 ua:0 ap:0 oos:235328
  10. [=======>............] sync'ed: 40.0% (235328/386240)K
  11. finish: 0:00:53 speed: 4,332 (4,192) K/sec
  12. 3: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
  13. ns:0 nr:119000 dw:119000 dr:0 al:0 bm:7 lo:1 pe:688 ua:0 ap:0 oos:17684
  14. [================>...] sync'ed: 88.3% (17684/136668)K
  15. finish: 0:00:03 speed: 4,600 (3,836) K/sec
看来,以后需要多注意DRBD状态,否则,Zenoss很有可能会切换到其它以住时期的运行状态。

11-22 22:21