Serious Performance Problems, Need help!!!
by Kumar, Amit
Dear Lustre,
We are having major performance problems this time and hard to grasp what is going on.
Health check all look good. Network looks good. But performance is bad.
(a) lfs df, output at the end of the email shows a couple of OST's temporarily unavailable, but this normally happens and it connects back. It does connect back but then it is unavailable again in a short while, this is repeating.
(b) Also included below are outputs for the following commands from every OSS
cat /proc/fs/lustre/devices
lctl get_param ost.*.ost_io.threads_max
lctl get_param ost.*.ost_io.threads_started
grep -i LBUG /var/log/messages
cat /proc/fs/lustre/health_check
cat /proc/sys/lnet/nis
(c)
(d) Based on this RPC stats that is attached to this email it seems a lot of pending pages to write to is probably causing this. Attached rpc_stats includes all ost's.
(e) Also the LNET peer stats below show a great deal of congestion with two of the OST.
I am not sure how to approach this in reducing the performance problems. Almost all OSS is seeing IO wait, backend storage also looks good.
Can anybody please advise on possible issue that may be causing this other than the file system being 88% full.
No changes were made to the system recently, except to refresh the disk in OST I deactivated OST temporarily while I was migrating the data off the deactivated OST. Since this problem this Friday I re-activated the deactivated OST, so that I could add additional OSS and OST to load balance, hence relieve the performance issues. It seemed to help a bit but not much.
Best,
Thank you,
Amit
Below here is the output of the following commands from each of the OSS.
cat /proc/fs/lustre/devices
lctl get_param ost.*.ost_io.threads_max
lctl get_param ost.*.ost_io.threads_started
grep -i LBUG /var/log/messages
cat /proc/fs/lustre/health_check
cat /proc/sys/lnet/nis
array2
0 UP mgc MGC10.1.1.40@tcp 87942af4-c7b4-5695-4680-2a3a4f232054 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST001a smuhpc-OST001a_UUID 439
3 UP obdfilter smuhpc-OST0000 smuhpc-OST0000_UUID 439
4 UP obdfilter smuhpc-OST0001 smuhpc-OST0001_UUID 439
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=367
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.51@tcp up -1 225 8 0 256 256 -1512
array2b
0 UP mgc MGC10.1.1.40@tcp f4072991-d501-f944-10b0-4c6a460c9c6d 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0002 smuhpc-OST0002_UUID 439
3 UP obdfilter smuhpc-OST0003 smuhpc-OST0003_UUID 438
4 UP obdfilter smuhpc-OST0008 smuhpc-OST0008_UUID 439
ost.OSS.ost_io.threads_max=128
ost.OSS.ost_io.threads_started=64
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.54@tcp up -1 225 8 0 256 256 -306
array3
0 UP mgc MGC10.1.1.40@tcp 524536bc-fb4f-bed5-6e55-924aa46112d1 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0004 smuhpc-OST0004_UUID 439
3 UP obdfilter smuhpc-OST0005 smuhpc-OST0005_UUID 439
4 UP obdfilter smuhpc-OST0006 smuhpc-OST0006_UUID 439
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=362
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.52@tcp up -1 225 8 0 256 256 -1037
array3b
0 UP mgc MGC10.1.1.40@tcp 00fdbef3-fd0c-18db-637b-eb869eb99309 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0007 smuhpc-OST0007_UUID 439
3 UP obdfilter smuhpc-OST0011 smuhpc-OST0011_UUID 439
4 UP obdfilter smuhpc-OST0012 smuhpc-OST0012_UUID 439
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=293
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.55@tcp up -1 225 8 0 256 256 -147
array4
0 UP mgc MGC10.1.1.40@tcp b90bd48b-3f2f-aa60-a1a4-e743ce1d4025 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST000b smuhpc-OST000b_UUID 439
3 UP obdfilter smuhpc-OST000c smuhpc-OST000c_UUID 439
4 UP obdfilter smuhpc-OST000d smuhpc-OST000d_UUID 439
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=512
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.53@tcp up -1 225 8 0 256 256 -966
array4b
0 UP mgc MGC10.1.1.40@tcp 1b31358c-ffc6-ca4d-14ea-78bf8804a15a 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST000e smuhpc-OST000e_UUID 439
3 UP obdfilter smuhpc-OST001c smuhpc-OST001c_UUID 437
4 UP obdfilter smuhpc-OST001d smuhpc-OST001d_UUID 437
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=512
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.56@tcp up -1 223 8 0 256 256 -655
array5
0 UP mgc MGC10.1.1.40@tcp 0bdb83f9-dbf5-aeaa-ff7d-66c0b6471811 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0009 smuhpc-OST0009_UUID 439
3 UP obdfilter smuhpc-OST000a smuhpc-OST000a_UUID 439
4 UP obdfilter smuhpc-OST000f smuhpc-OST000f_UUID 439
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=512
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.57@tcp up -1 225 8 0 256 256 -385
array5b
0 UP mgc MGC10.1.1.40@tcp d5982303-1e80-3ba5-c88b-b712e2d7c7af 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0010 smuhpc-OST0010_UUID 439
3 UP obdfilter smuhpc-OST0017 smuhpc-OST0017_UUID 439
4 UP obdfilter smuhpc-OST001b smuhpc-OST001b_UUID 439
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=312
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.58@tcp up -1 225 8 0 256 249 -383
array6
0 UP mgc MGC10.1.1.40@tcp 624e193a-3f28-2936-14e8-a3ff130bcd0f 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0030 smuhpc-OST0030_UUID 436
3 UP obdfilter smuhpc-OST0031 smuhpc-OST0031_UUID 436
4 UP obdfilter smuhpc-OST0032 smuhpc-OST0032_UUID 436
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=128
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.59@tcp up -1 225 8 0 256 238 -475
array6b
0 UP mgc MGC10.1.1.40@tcp 1854af7c-31b9-43c5-058c-4953afb936bb 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0033 smuhpc-OST0033_UUID 436
3 UP obdfilter smuhpc-OST0034 smuhpc-OST0034_UUID 436
4 UP obdfilter smuhpc-OST0035 smuhpc-OST0035_UUID 436
ost.OSS.ost_io.threads_max=256
ost.OSS.ost_io.threads_started=128
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.60@tcp up -1 225 8 0 256 239 -217
array8
0 UP mgc MGC10.1.1.40@tcp a6744840-8c1a-cd8a-487b-db2e9efbf856 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0013 smuhpc-OST0013_UUID 439
3 UP obdfilter smuhpc-OST0014 smuhpc-OST0014_UUID 439
4 UP obdfilter smuhpc-OST0015 smuhpc-OST0015_UUID 439
5 UP obdfilter smuhpc-OST0016 smuhpc-OST0016_UUID 439
6 UP obdfilter smuhpc-OST0018 smuhpc-OST0018_UUID 439
7 UP obdfilter smuhpc-OST0019 smuhpc-OST0019_UUID 439
ost.OSS.ost_io.threads_max=512
ost.OSS.ost_io.threads_started=512
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.62@tcp up -1 225 8 0 256 256 -673
array7
0 UP mgc MGC10.1.1.40@tcp 315ffeaf-3075-24d7-2a01-02e062b60e34 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter smuhpc-OST0036 smuhpc-OST0036_UUID 437
3 UP obdfilter smuhpc-OST0037 smuhpc-OST0037_UUID 437
4 UP obdfilter smuhpc-OST0038 smuhpc-OST0038_UUID 437
5 UP obdfilter smuhpc-OST0039 smuhpc-OST0039_UUID 437
6 UP obdfilter smuhpc-OST003a smuhpc-OST003a_UUID 437
7 UP obdfilter smuhpc-OST003b smuhpc-OST003b_UUID 437
ost.OSS.ost_io.threads_max=256
ost.OSS.ost_io.threads_started=64
healthy
nid status alive refs peer rtr max tx min
0@lo up 0 2 0 0 0 0 0
10.1.1.61@tcp up -1 225 8 0 256 251 -1421
MGS/MDS_NODE# cat /proc/sys/lnet/peers | grep "10\.1\.1\." (below here are our ost's, and you see congestion on two of those, although the health_check shows healthy)
10.1.1.51@tcp 1 up 8 8 8 8 -6732 0
10.1.1.52@tcp 1 up 8 8 8 8 -2753 0
10.1.1.53@tcp 1 up 8 8 8 8 -4 0
10.1.1.54@tcp 1 up 8 8 8 8 -40 0
10.1.1.55@tcp 1 up 8 8 8 8 -7 0
10.1.1.56@tcp 1 up 8 8 8 8 0 0
10.1.1.57@tcp 1 up 8 8 8 8 -4 0
10.1.1.58@tcp 1 up 8 8 8 8 -6 0
10.1.1.59@tcp 1 up 8 8 8 8 -2 0
10.1.1.60@tcp 1 up 8 8 8 8 -1 0
10.1.1.61@tcp 1 up 8 8 8 8 -15 0
10.1.1.62@tcp 1 up 8 8 8 8 -11 0
=======More LOGS from MDS/MGS ====
# grep '[0-9]' /proc/fs/lustre/osc/*/kbytes{free,avail,total}
/proc/fs/lustre/osc/smuhpc-OST0000-osc/kbytesfree:514058156
/proc/fs/lustre/osc/smuhpc-OST0001-osc/kbytesfree:765667120
/proc/fs/lustre/osc/smuhpc-OST0002-osc/kbytesfree:1096019280
grep: /proc/fs/lustre/osc/smuhpc-OST0003-osc/kbytesfree: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0004-osc/kbytesfree:1577637660
/proc/fs/lustre/osc/smuhpc-OST0005-osc/kbytesfree:132305164
/proc/fs/lustre/osc/smuhpc-OST0006-osc/kbytesfree:899697048
/proc/fs/lustre/osc/smuhpc-OST0007-osc/kbytesfree:857944436
/proc/fs/lustre/osc/smuhpc-OST0008-osc/kbytesfree:36161928
/proc/fs/lustre/osc/smuhpc-OST0009-osc/kbytesfree:39061480
/proc/fs/lustre/osc/smuhpc-OST000a-osc/kbytesfree:938678228
/proc/fs/lustre/osc/smuhpc-OST000b-osc/kbytesfree:8604452
/proc/fs/lustre/osc/smuhpc-OST000c-osc/kbytesfree:44878900
/proc/fs/lustre/osc/smuhpc-OST000d-osc/kbytesfree:1117771508
/proc/fs/lustre/osc/smuhpc-OST000e-osc/kbytesfree:769454268
/proc/fs/lustre/osc/smuhpc-OST000f-osc/kbytesfree:56939372
/proc/fs/lustre/osc/smuhpc-OST0010-osc/kbytesfree:210416704
/proc/fs/lustre/osc/smuhpc-OST0011-osc/kbytesfree:1315953944
/proc/fs/lustre/osc/smuhpc-OST0012-osc/kbytesfree:1112498952
/proc/fs/lustre/osc/smuhpc-OST0013-osc/kbytesfree:917528092
/proc/fs/lustre/osc/smuhpc-OST0014-osc/kbytesfree:818228736
/proc/fs/lustre/osc/smuhpc-OST0015-osc/kbytesfree:119717344
/proc/fs/lustre/osc/smuhpc-OST0016-osc/kbytesfree:818664044
/proc/fs/lustre/osc/smuhpc-OST0017-osc/kbytesfree:1307525340
/proc/fs/lustre/osc/smuhpc-OST0018-osc/kbytesfree:561629216
/proc/fs/lustre/osc/smuhpc-OST0019-osc/kbytesfree:682050424
/proc/fs/lustre/osc/smuhpc-OST001a-osc/kbytesfree:1262541880
/proc/fs/lustre/osc/smuhpc-OST001b-osc/kbytesfree:864048788
/proc/fs/lustre/osc/smuhpc-OST001c-osc/kbytesfree:511371988
/proc/fs/lustre/osc/smuhpc-OST001d-osc/kbytesfree:109860844
grep: /proc/fs/lustre/osc/smuhpc-OST0030-osc/kbytesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0031-osc/kbytesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0032-osc/kbytesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0033-osc/kbytesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0034-osc/kbytesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0035-osc/kbytesfree: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0036-osc/kbytesfree:718292640
/proc/fs/lustre/osc/smuhpc-OST0037-osc/kbytesfree:472531244
/proc/fs/lustre/osc/smuhpc-OST0038-osc/kbytesfree:433755684
/proc/fs/lustre/osc/smuhpc-OST0039-osc/kbytesfree:875580388
/proc/fs/lustre/osc/smuhpc-OST003a-osc/kbytesfree:1161276948
grep: /proc/fs/lustre/osc/smuhpc-OST003b-osc/kbytesfree: Resource temporarily unavailable
/proc/fs/lustre/osc/smuhpc-OST0000-osc/kbytesavail:514033840
/proc/fs/lustre/osc/smuhpc-OST0001-osc/kbytesavail:765639756
/proc/fs/lustre/osc/smuhpc-OST0002-osc/kbytesavail:1095950892
grep: /proc/fs/lustre/osc/smuhpc-OST0003-osc/kbytesavail: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0004-osc/kbytesavail:1577629868
/proc/fs/lustre/osc/smuhpc-OST0005-osc/kbytesavail:132295072
/proc/fs/lustre/osc/smuhpc-OST0006-osc/kbytesavail:899689368
/proc/fs/lustre/osc/smuhpc-OST0007-osc/kbytesavail:857942648
/proc/fs/lustre/osc/smuhpc-OST0008-osc/kbytesavail:36140876
/proc/fs/lustre/osc/smuhpc-OST0009-osc/kbytesavail:38998500
/proc/fs/lustre/osc/smuhpc-OST000a-osc/kbytesavail:938670344
/proc/fs/lustre/osc/smuhpc-OST000b-osc/kbytesavail:8593840
/proc/fs/lustre/osc/smuhpc-OST000c-osc/kbytesavail:44876596
/proc/fs/lustre/osc/smuhpc-OST000d-osc/kbytesavail:1117758504
/proc/fs/lustre/osc/smuhpc-OST000e-osc/kbytesavail:769447360
/proc/fs/lustre/osc/smuhpc-OST000f-osc/kbytesavail:56922292
/proc/fs/lustre/osc/smuhpc-OST0010-osc/kbytesavail:210406920
/proc/fs/lustre/osc/smuhpc-OST0011-osc/kbytesavail:1315948464
/proc/fs/lustre/osc/smuhpc-OST0012-osc/kbytesavail:1112487208
/proc/fs/lustre/osc/smuhpc-OST0013-osc/kbytesavail:917520972
/proc/fs/lustre/osc/smuhpc-OST0014-osc/kbytesavail:818200064
/proc/fs/lustre/osc/smuhpc-OST0015-osc/kbytesavail:119708876
/proc/fs/lustre/osc/smuhpc-OST0016-osc/kbytesavail:818659948
/proc/fs/lustre/osc/smuhpc-OST0017-osc/kbytesavail:1307516124
/proc/fs/lustre/osc/smuhpc-OST0018-osc/kbytesavail:561624584
/proc/fs/lustre/osc/smuhpc-OST0019-osc/kbytesavail:682045540
/proc/fs/lustre/osc/smuhpc-OST001a-osc/kbytesavail:1262529492
/proc/fs/lustre/osc/smuhpc-OST001b-osc/kbytesavail:863983524
/proc/fs/lustre/osc/smuhpc-OST001c-osc/kbytesavail:511362064
/proc/fs/lustre/osc/smuhpc-OST001d-osc/kbytesavail:109827908
grep: /proc/fs/lustre/osc/smuhpc-OST0030-osc/kbytesavail: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0031-osc/kbytesavail: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0032-osc/kbytesavail: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0033-osc/kbytesavail: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0034-osc/kbytesavail: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0035-osc/kbytesavail: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0036-osc/kbytesavail:718253728
/proc/fs/lustre/osc/smuhpc-OST0037-osc/kbytesavail:472467152
/proc/fs/lustre/osc/smuhpc-OST0038-osc/kbytesavail:433729872
/proc/fs/lustre/osc/smuhpc-OST0039-osc/kbytesavail:875578332
/proc/fs/lustre/osc/smuhpc-OST003a-osc/kbytesavail:1161272852
grep: /proc/fs/lustre/osc/smuhpc-OST003b-osc/kbytesavail: Resource temporarily unavailable
/proc/fs/lustre/osc/smuhpc-OST0000-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST0001-osc/kbytestotal:9612387536
/proc/fs/lustre/osc/smuhpc-OST0002-osc/kbytestotal:11534862728
grep: /proc/fs/lustre/osc/smuhpc-OST0003-osc/kbytestotal: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0004-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST0005-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST0006-osc/kbytestotal:9612387536
/proc/fs/lustre/osc/smuhpc-OST0007-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST0008-osc/kbytestotal:9615574536
/proc/fs/lustre/osc/smuhpc-OST0009-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST000a-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST000b-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST000c-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST000d-osc/kbytestotal:9612387536
/proc/fs/lustre/osc/smuhpc-OST000e-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST000f-osc/kbytestotal:9612387536
/proc/fs/lustre/osc/smuhpc-OST0010-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST0011-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST0012-osc/kbytestotal:9615574536
/proc/fs/lustre/osc/smuhpc-OST0013-osc/kbytestotal:13452678016
/proc/fs/lustre/osc/smuhpc-OST0014-osc/kbytestotal:13452678016
/proc/fs/lustre/osc/smuhpc-OST0015-osc/kbytestotal:11530866816
/proc/fs/lustre/osc/smuhpc-OST0016-osc/kbytestotal:13452678016
/proc/fs/lustre/osc/smuhpc-OST0017-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST0018-osc/kbytestotal:13452678016
/proc/fs/lustre/osc/smuhpc-OST0019-osc/kbytestotal:11530866816
/proc/fs/lustre/osc/smuhpc-OST001a-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST001b-osc/kbytestotal:9615574536
/proc/fs/lustre/osc/smuhpc-OST001c-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST001d-osc/kbytestotal:9615574536
grep: /proc/fs/lustre/osc/smuhpc-OST0030-osc/kbytestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0031-osc/kbytestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0032-osc/kbytestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0033-osc/kbytestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0034-osc/kbytestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0035-osc/kbytestotal: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0036-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST0037-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST0038-osc/kbytestotal:11534862728
/proc/fs/lustre/osc/smuhpc-OST0039-osc/kbytestotal:11538687128
/proc/fs/lustre/osc/smuhpc-OST003a-osc/kbytestotal:9612387536
grep: /proc/fs/lustre/osc/smuhpc-OST003b-osc/kbytestotal: Resource temporarily unavailable
# grep '[0-9]' /proc/fs/lustre/osc/*/files{free,total}
/proc/fs/lustre/osc/smuhpc-OST0000-osc/filesfree:128514539
/proc/fs/lustre/osc/smuhpc-OST0001-osc/filesfree:191416790
/proc/fs/lustre/osc/smuhpc-OST0002-osc/filesfree:274004820
grep: /proc/fs/lustre/osc/smuhpc-OST0003-osc/filesfree: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0004-osc/filesfree:394395591
/proc/fs/lustre/osc/smuhpc-OST0005-osc/filesfree:33076291
/proc/fs/lustre/osc/smuhpc-OST0006-osc/filesfree:224911717
/proc/fs/lustre/osc/smuhpc-OST0007-osc/filesfree:214486110
/proc/fs/lustre/osc/smuhpc-OST0008-osc/filesfree:8856919
/proc/fs/lustre/osc/smuhpc-OST0009-osc/filesfree:9624045
/proc/fs/lustre/osc/smuhpc-OST000a-osc/filesfree:234669553
/proc/fs/lustre/osc/smuhpc-OST000b-osc/filesfree:2151113
/proc/fs/lustre/osc/smuhpc-OST000c-osc/filesfree:11219725
/proc/fs/lustre/osc/smuhpc-OST000d-osc/filesfree:279442892
/proc/fs/lustre/osc/smuhpc-OST000e-osc/filesfree:192357679
/proc/fs/lustre/osc/smuhpc-OST000f-osc/filesfree:14234843
/proc/fs/lustre/osc/smuhpc-OST0010-osc/filesfree:52604176
/proc/fs/lustre/osc/smuhpc-OST0011-osc/filesfree:328988486
/proc/fs/lustre/osc/smuhpc-OST0012-osc/filesfree:278118850
/proc/fs/lustre/osc/smuhpc-OST0013-osc/filesfree:229382023
/proc/fs/lustre/osc/smuhpc-OST0014-osc/filesfree:204557180
/proc/fs/lustre/osc/smuhpc-OST0015-osc/filesfree:29929336
/proc/fs/lustre/osc/smuhpc-OST0016-osc/filesfree:204663451
/proc/fs/lustre/osc/smuhpc-OST0017-osc/filesfree:326881334
/proc/fs/lustre/osc/smuhpc-OST0018-osc/filesfree:140407304
/proc/fs/lustre/osc/smuhpc-OST0019-osc/filesfree:170512603
/proc/fs/lustre/osc/smuhpc-OST001a-osc/filesfree:315635470
/proc/fs/lustre/osc/smuhpc-OST001b-osc/filesfree:216012197
/proc/fs/lustre/osc/smuhpc-OST001c-osc/filesfree:127842996
/proc/fs/lustre/osc/smuhpc-OST001d-osc/filesfree:27465211
grep: /proc/fs/lustre/osc/smuhpc-OST0030-osc/filesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0031-osc/filesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0032-osc/filesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0033-osc/filesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0034-osc/filesfree: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0035-osc/filesfree: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0036-osc/filesfree:179276300
/proc/fs/lustre/osc/smuhpc-OST0037-osc/filesfree:118132811
/proc/fs/lustre/osc/smuhpc-OST0038-osc/filesfree:108438921
/proc/fs/lustre/osc/smuhpc-OST0039-osc/filesfree:218891001
/proc/fs/lustre/osc/smuhpc-OST003a-osc/filesfree:290319237
grep: /proc/fs/lustre/osc/smuhpc-OST003b-osc/filesfree: Resource temporarily unavailable
/proc/fs/lustre/osc/smuhpc-OST0000-osc/filestotal:132250935
/proc/fs/lustre/osc/smuhpc-OST0001-osc/filestotal:193743661
/proc/fs/lustre/osc/smuhpc-OST0002-osc/filestotal:277776374
grep: /proc/fs/lustre/osc/smuhpc-OST0003-osc/filestotal: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0004-osc/filestotal:396440189
/proc/fs/lustre/osc/smuhpc-OST0005-osc/filestotal:35044722
/proc/fs/lustre/osc/smuhpc-OST0006-osc/filestotal:226783057
/proc/fs/lustre/osc/smuhpc-OST0007-osc/filestotal:217558056
/proc/fs/lustre/osc/smuhpc-OST0008-osc/filestotal:11184523
/proc/fs/lustre/osc/smuhpc-OST0009-osc/filestotal:12231803
/proc/fs/lustre/osc/smuhpc-OST000a-osc/filestotal:237327760
/proc/fs/lustre/osc/smuhpc-OST000b-osc/filestotal:4003238
/proc/fs/lustre/osc/smuhpc-OST000c-osc/filestotal:12981815
/proc/fs/lustre/osc/smuhpc-OST000d-osc/filestotal:281106176
/proc/fs/lustre/osc/smuhpc-OST000e-osc/filestotal:195190328
/proc/fs/lustre/osc/smuhpc-OST000f-osc/filestotal:16476504
/proc/fs/lustre/osc/smuhpc-OST0010-osc/filestotal:55089005
/proc/fs/lustre/osc/smuhpc-OST0011-osc/filestotal:330925776
/proc/fs/lustre/osc/smuhpc-OST0012-osc/filestotal:279931713
/proc/fs/lustre/osc/smuhpc-OST0013-osc/filestotal:231994647
/proc/fs/lustre/osc/smuhpc-OST0014-osc/filestotal:206633272
/proc/fs/lustre/osc/smuhpc-OST0015-osc/filestotal:31825125
/proc/fs/lustre/osc/smuhpc-OST0016-osc/filestotal:206716377
/proc/fs/lustre/osc/smuhpc-OST0017-osc/filestotal:329458681
/proc/fs/lustre/osc/smuhpc-OST0018-osc/filestotal:142450211
/proc/fs/lustre/osc/smuhpc-OST0019-osc/filestotal:172358875
/proc/fs/lustre/osc/smuhpc-OST001a-osc/filestotal:318295996
/proc/fs/lustre/osc/smuhpc-OST001b-osc/filestotal:218409008
/proc/fs/lustre/osc/smuhpc-OST001c-osc/filestotal:129660363
/proc/fs/lustre/osc/smuhpc-OST001d-osc/filestotal:29250131
grep: /proc/fs/lustre/osc/smuhpc-OST0030-osc/filestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0031-osc/filestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0032-osc/filestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0033-osc/filestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0034-osc/filestotal: Cannot send after transport endpoint shutdown
grep: /proc/fs/lustre/osc/smuhpc-OST0035-osc/filestotal: Cannot send after transport endpoint shutdown
/proc/fs/lustre/osc/smuhpc-OST0036-osc/filestotal:180363444
/proc/fs/lustre/osc/smuhpc-OST0037-osc/filestotal:119215177
/proc/fs/lustre/osc/smuhpc-OST0038-osc/filestotal:109491390
/proc/fs/lustre/osc/smuhpc-OST0039-osc/filestotal:220028416
/proc/fs/lustre/osc/smuhpc-OST003a-osc/filestotal:291302059
grep: /proc/fs/lustre/osc/smuhpc-OST003b-osc/filestotal: Resource temporarily unavailable
# grep '[0-9]' /proc/fs/lustre/mds/*/kbytes{free,avail,total}
/proc/fs/lustre/mds/smuhpc-MDT0000/kbytesfree:677031612
/proc/fs/lustre/mds/smuhpc-MDT0000/kbytesavail:677031612
/proc/fs/lustre/mds/smuhpc-MDT0000/kbytestotal:688076544
# grep '[0-9]' /proc/fs/lustre/mds/*/files{free,total}
/proc/fs/lustre/mds/smuhpc-MDT0000/filesfree:154564006
/proc/fs/lustre/mds/smuhpc-MDT0000/filestotal:196608000
# lfs df
UUID 1K-blocks Used Available Use% Mounted on
smuhpc-MDT0000_UUID 688076544 11044932 677031612 2% /lustre[MDT:0]
smuhpc-OST0000_UUID 11538687128 11024628972 514033840 96% /lustre[OST:0]
smuhpc-OST0001_UUID 9612387536 8846797188 765561148 92% /lustre[OST:1]
smuhpc-OST0002_UUID 11534862728 10438969420 1095808024 90% /lustre[OST:2]
smuhpc-OST0003_UUID 11538687128 10237552284 1301088824 89% /lustre[OST:3]
smuhpc-OST0004_UUID 11534862728 9957280360 1577573564 86% /lustre[OST:4]
smuhpc-OST0005_UUID 11538687128 11406381964 132295072 99% /lustre[OST:5]
smuhpc-OST0006_UUID 9612387536 8712821564 899477908 91% /lustre[OST:6]
smuhpc-OST0007_UUID 11534862728 10676918312 857941880 93% /lustre[OST:7]
smuhpc-OST0008_UUID 9615574536 9587388940 28150780 100% /lustre[OST:8]
smuhpc-OST0009_UUID 11534862728 11502157276 32703404 100% /lustre[OST:9]
smuhpc-OST000a_UUID 11538687128 10600008916 938671044 92% /lustre[OST:10]
smuhpc-OST000b_UUID 11534862728 11526258276 8593840 100% /lustre[OST:11]
smuhpc-OST000c_UUID 11538687128 11493808228 44876596 100% /lustre[OST:12]
smuhpc-OST000d_UUID 9612387536 8494655908 1117723728 88% /lustre[OST:13]
smuhpc-OST000e_UUID 11534862728 10765432012 769356988 93% /lustre[OST:14]
smuhpc-OST000f_UUID 9612387536 9555448164 56922292 99% /lustre[OST:15]
smuhpc-OST0010_UUID 11534862728 11324446024 210408036 98% /lustre[OST:16]
smuhpc-OST0011_UUID 11538687128 10222777216 1315902940 89% /lustre[OST:17]
smuhpc-OST0012_UUID 9615574536 8503099372 1112462716 88% /lustre[OST:18]
smuhpc-OST0013_UUID 13452678016 12535227748 917441212 93% /lustre[OST:19]
smuhpc-OST0014_UUID 13452678016 12634464700 818178500 94% /lustre[OST:20]
smuhpc-OST0015_UUID 11530866816 11411149472 119708876 99% /lustre[OST:21]
smuhpc-OST0016_UUID 13452678016 12639406780 813217988 94% /lustre[OST:22]
smuhpc-OST0017_UUID 11538687128 10231302100 1307378884 89% /lustre[OST:23]
smuhpc-OST0018_UUID 13452678016 12891084320 561588224 96% /lustre[OST:24]
smuhpc-OST0019_UUID 11530866816 10848816880 682044044 94% /lustre[OST:25]
smuhpc-OST001a_UUID 11534862728 10272389460 1262461836 89% /lustre[OST:26]
smuhpc-OST001b_UUID 9615574536 8751547300 864019044 91% /lustre[OST:27]
smuhpc-OST001c_UUID 11538687128 11027353036 511322908 96% /lustre[OST:28]
smuhpc-OST001d_UUID 9615574536 9505741344 109832588 99% /lustre[OST:29]
smuhpc-OST0030_UUID 11534862728 7027461656 4507389808 61% /lustre[OST:48]
smuhpc-OST0031_UUID 11538687128 2208373512 9330099084 19% /lustre[OST:49]
smuhpc-OST0032_UUID 9612387536 5795054380 3817314724 60% /lustre[OST:50]
smuhpc-OST0033_UUID 11534862728 7414666856 4120177440 64% /lustre[OST:51]
smuhpc-OST0034_UUID 11538687128 7489405512 4049271072 65% /lustre[OST:52]
smuhpc-OST0035_UUID 9615574536 6709760396 2905811696 70% /lustre[OST:53]
smuhpc-OST0036_UUID 11534862728 10824985124 709871280 94% /lustre[OST:54]
smuhpc-OST0037_UUID : Resource temporarily unavailable
smuhpc-OST0038_UUID : Resource temporarily unavailable
smuhpc-OST0039_UUID 11538687128 10663152820 875526472 92% /lustre[OST:57]
smuhpc-OST003a_UUID 9612387536 8451144380 1161159188 88% /lustre[OST:58]
smuhpc-OST003b_UUID 9615574536 9396687868 218849404 98% /lustre[OST:59]
filesystem summary: 446049266544 393606006040 52442216896 88% /lustre
7 years, 10 months
Where is cfs_spin_unlock?
by Jay Lan
I tried to build lustre-2.4.0 client for sles11sp2 3.0.74-0.6.6.2.
The build failed because cfs_spin_unlock was not defined:
/usr/src/packages/BUILD/lustre-2.4.0/lnet/klnds/socklnd/socklnd_cb.c: In
function 'ksocknal_check_peer_timeouts':
/usr/src/packages/BUILD/lustre-2.4.0/lnet/klnds/socklnd/socklnd_cb.c:2525:
error: implicit declaration of function 'cfs_spin_unlock'
Where is that defined?
Thanks,
Jay
7 years, 10 months
Upgrading from Lustre 1.8.4 to 2.1.6
by Bob Ball
Hi, all.
After running for a good long time with Lustre 1.8.4 everywhere on SL5
we are in the process of upgrading everywhere to SL6.4. We plan on
taking down our entire Lustre system at once, then bringing it back up
under SL6.4/Lustre 2.1.6. The process as documented in Chapter 16 of
the user's guide looks pretty straight forward. Are there any "gotchas"
I should know about? We are not planning on reformatting any file systems.
Thanks,
bob
PS, apologies to whom it may concern that I originally sent this to
hpdd-discuss-request
7 years, 10 months
Lustre Classes in July
by Ferber, Dan
Intel's High Performance Data Division is pleased to announce two Lustre* training opportunities; one in England July 22-24 and the other in South Korea July 15-17.
These are three day lecture and lab hands on opportunities for learning how Install and Administer Lustre.
For more information and to sign up to attend the classes, see:
http://www.whamcloud.com/training/
Best regards,
Dan Ferber
*Other names and brands may be claimed as the property of others
------
Dan Ferber
Sr. Manager, Systems Engineering and Product Management
Intel® High Performance Data Division
Office: +1 651.344.1846<tel:%2B1%20651.344.1846>
Mobile: +1 651.344.1846<tel:%2B1%20651.344.1846>
dan.ferber(a)intel.com<mailto:dan.ferber@intel.com>
7 years, 10 months
Re: [HPDD-discuss] [cdwg] Alternative non accelerated checksums
by Dilger, Andreas
On 2013-06-21, at 8:34, "James A Simmons" <uja(a)ornl.gov> wrote:
> Recently I have been doing an evaluation of the performance of check
> summing in the Lustre code base. From the results you can see it can be
> very expensive when you lack hardware acceleration. Unfortunately
> we have some platforms were their is a high cost but check summing
> is a requirement.
It is also important to consider what checksum algorithm is used on the server, since the server will have to do the checksumming for all of the clients.
> Besides the core algorithms I have added a few of my own to
> see how they measure up. We have csum which is your normal IP Header
> check sum.
The IP header checksum is only 16-bit, so it is only suitable for a small amount of data. It is definitely not suitable for 1MB or 4MB RPC sizes.
> For the non cryptographic hashes it's the IP check sum and murmur3
> that does the best. This version of murmur3 only generates 32 check
> sums but their exist a 128 bit version that is suppose to be faster.
> It could be worth while to explore. The IP check sum from the linux
> kernel is assembly optimized but my additional algorithms are generic
> C.
You should test with the kernel cryptoapi code, since AFAIK there are assembly versions of the common algorithms already. Check out how the libcfs code is already handling the crc32 code - it benchmarks each algorithm at startup and dumps the results in the Lustre debug log.
> The final question is the Lustre community interested in the new
> algorithms? If so I can push forward that work.
I'm not against it if there are significant improvements to be had.
It surprises me that newer CPUs do not have hardware-accelerated checksums of some sort. Is it just that the assembly versions have not been implemented in the kernels that Lustre is running on? Could they be implemented in libcfs as was done with crc32 and then submitted to the upstream kernel (so everyone benefits and we don't have to maintain them forever)?
Cheers, Andreas
7 years, 10 months
Lustre Error
by Kumar, Amit
Dear Lustre,
I am seeing quite a bit of these errors on quite a number of lustre clients. Any idea what could be causing many of the client to have connection issues?
Network seems to be solid not issues on that end, because it is internal.
LustreError: 11-0: an error occurred while communicating with 10.1.1.58@tcp. The ost_connect operation failed with -114
LustreError: Skipped 19 previous similar messages
Lustre: 5315:0:(import.c:517:import_select_connection()) smuhpc-OST0017-osc-ffff810c6d349400: tried all connections, increasing latency to 25s
Lustre: 5315:0:(import.c:517:import_select_connection()) Skipped 20 previous similar messages
Lustre: 5314:0:(client.c:1496:ptlrpc_expire_one_request()) @@@ Request x1437061797490383 sent from smuhpc-OST0017-osc-ffff810c6d349400 to NID 10.1.1.58@tcp 30s ago has timed out (30s prior to deadline).
req@ffff81045d656c00 x1437061797490383/t0 o8->smuhpc-OST0017_UUID@10.1.1.58@tcp:28/4 lens 368/584 e 0 to 1 dl 1371822883 ref 1 fl Rpc:N/0/0 rc 0/0
Lustre: 5314:0:(client.c:1496:ptlrpc_expire_one_request()) @@@ Request x1437061797492276 sent from smuhpc-OST0017-osc-ffff810c6d349400 to NID 10.1.1.58@tcp 20s ago has timed out (20s prior to deadline).
req@ffff81067e7b4800 x1437061797492276/t0 o8->smuhpc-OST0017_UUID@10.1.1.58@tcp:28/4 lens 368/584 e 0 to 1 dl 1371823117 ref 1 fl Rpc:N/0/0 rc 0/0
LustreError: 11-0: an error occurred while communicating with 10.1.1.58@tcp. The ost_connect operation failed with -114
LustreError: Skipped 25 previous similar messages
Any thoughts will be greatly appreciated.
Best,
Amit
7 years, 10 months
Wondering about running OSTs on my cloud platform.
by Kevin Thorpe
Hi, we've been looking at private (and maybe hybrid) cloud provisioning.
We've got
an architecture for the traditional SAN + hosts model but we're interested
in the
SANless SAN devices that Nutanix offer. We realise that a hardware SAN
device
is more likely to deliver performance but we think we could get away with
using
distributed local storage on the VM host servers.
Do you think that running an OST VM on each of our VM hosts utilising local
disks
would be a sensible thing to do? I realise this would eat into RAM and
processor
on each host, but new hosts would be far cheaper than a SAN device. I'm
particularly
interested in the ZFS option so se get SSD cache on each node. The only
thing
that I can see that Nutanix do that Lustre can't is migrate data to the
nodes using
that data, or am I wrong on that?
--
Kevin Thorpe
Chief Technical Officer
PI Benchmark
7 years, 10 months
Alternative non accelerated checksums
by James A Simmons
Hello
Recently I have been doing an evaluation of the performance of
check
summing in the Lustre code base. From the results you can see it can be
very expensive when you lack hardware acceleration. Unfortunately
we have some platforms were their is a high cost but check summing
is a requirement. Since the best case for non accelerated is not
good enough I have looked into other hashing type algorithms to see
if they can meet the communities needs.
Besides the core algorithms I have added a few of my own to
see how they measure up. We have csum which is your normal IP Header
check sum. A 32 bit version of murmur3 was implemented. I used the
jhash.h from linux to implement jenkins and lastly siphash was
implemented. Siphash author claims it to be more secure algorithm
in the same category as md5 but with much better speeds. I have tested
against four systems. The first set of test is for non cryptographic
hashes and the second is for the cryptographic set.
AMD Opteron(tm) Processor 6168 @ 1.9GHz stepping 1
Lustre: Crypto hash algorithm adler32 speed = 754 MB/s
Lustre: Crypto hash algorithm csum speed = 1834 MB/s
Lustre: Crypto hash algorithm murmur speed = 1093 MB/s
Lustre: Crypto hash algorithm jenkins speed = 763 MB/s
Lustre: Crypto hash algorithm md5 speed = 306 MB/s
Lustre: Crypto hash algorithm sha1 speed = 123 MB/s
Lustre: Crypto hash algorithm siphash speed = 593 MB/s
AMD Opteron(TM) Processor 6274 @ 2.2GHz stepping 2
Lustre: Crypto hash algorithm adler32 speed = 670 MB/s
Lustre: Crypto hash algorithm csum speed = 802 MB/s
Lustre: Crypto hash algorithm murmur speed = 1511 MB/s
Lustre: Crypto hash algorithm jenkins speed = 614 MB/s
Lustre: Crypto hash algorithm md5 speed = 309 MB/s
Lustre: Crypto hash algorithm sha1 speed = 118 MB/s
Lustre: Crypto hash algorithm siphash speed = 669 MB/s
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz stepping 05
Lustre: Crypto hash algorithm adler32 speed = 836 MB/s
Lustre: Crypto hash algorithm csum speed = 3639 MB/s
Lustre: Crypto hash algorithm murmur speed = 1261 MB/s
Lustre: Crypto hash algorithm jenkins speed = 766 MB/s
Lustre: Crypto hash algorithm md5 speed = 265 MB/s
Lustre: Crypto hash algorithm sha1 speed = 108 MB/s
Lustre: Crypto hash algorithm sha256 speed = 75 MB/s
Lustre: Crypto hash algorithm sha384 speed = 105 MB/s
Lustre: Crypto hash algorithm sha512 speed = 105 MB/s
Lustre: Crypto hash algorithm siphash speed = 570 MB/s
Intel(R) Xeon(R) CPU E5-2603 0 @ 1.80GHz stepping 07
Lustre: Crypto hash algorithm adler32 speed = 1476 MB/s
Lustre: Crypto hash algorithm csum speed = 1769 MB/s
Lustre: Crypto hash algorithm murmur speed = 1243 MB/s
Lustre: Crypto hash algorithm jenkins speed = 956 MB/s
Lustre: Crypto hash algorithm md5 speed = 277 MB/s
Lustre: Crypto hash algorithm sha1 speed = 118 MB/s
Lustre: Crypto hash algorithm sha256 speed = 75 MB/s
Lustre: Crypto hash algorithm sha384 speed = 112 MB/s
Lustre: Crypto hash algorithm sha512 speed = 114 MB/s
Lustre: Crypto hash algorithm siphash speed = 562 MB/s
>From the data so far you can see when you compare siphash to shaX and
md5 it is a clear winning in performance for the cryptographic hashes.
For the non cryptographic hashes it's the IP check sum and murmur3
that does the best. This version of murmur3 only generates 32 check
sums but their exist a 128 bit version that is suppose to be faster.
It could be worth while to explore. The IP check sum from the linux
kernel is assembly optimized but my additional algorithms are generic
C. If done right we could speed up murmur3.
The final question is the Lustre community interested in the new
algorithms? If so I can push forward that work.
7 years, 10 months
