The current Lustre manual recommends using Pacemaker with Lustre. However, it does not provide detailed examples. It provides a link to a wiki site, http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre, which describes the process, but still does not give any specific examples.
Does anyone have config files for Pacemaker and Corosync that they'd be willing to share?
Recently the question was asked if we could enable file locking on our lustre file system (because of automake/autoconf). We are currently running lustre 1.8.8 on our mds/loss servers, which apparently doesn't support (or it isn't enabled) file locking.
Is this something that could be done, non-destructively, to the current system... or would we have to upgrade to a newer version?
Kurt J. Strosahl
Scientific Computing Group, Thomas Jefferson National Accelerator Facility
I have setup 1.8 on one MDT and 4 OSS
Recently the MDT crashed and have to recover from backup, then filesystem
is mountable but cannot write onto the filesystem (probably due to
inconsistency between MDT and OSS)
Pls, can anyone give some advice, thanks.
Here is an update on the Lustre 2.5 release.
-A number of landings made - see http://git.whamcloud.com/?p=fs/lustre-release.git;a=shortlog;h=refs/heads...
-Now that the Feature Freeze has passed, the full list of features in 2.5 is known
Xattr Cache (LU-2869)
LNET router improvements (LU-2934/2950)
Errno translation (step towards sparc support) (LU-2743)
Permanent Parameters option for lctl set_param (LU-3155)
Reduce ldlm_poold cpu utilization (LU-2924)
HSM (LU-3608) There is still some remaining work for this to be feature complete. As per the discussion on the CDWG call on Wednesday we are going to see we have extended the landing window for these patches only in view of the high interest in this feature and the belief that this work is close to completion. I will send out another update early next week with the final outcome on this.
* POSIX Copy tool http://review.whamcloud.com/#/c/4737/
* Disaster Recovery http://review.whamcloud.com/#/c/7027/
* Coordinator http://review.whamcloud.com/#/c/6912/
- Testing on the 2.4.52 tag is complete; testing on the 2.4.53 tag is ongoing
-Full list available at https://jira.hpdd.intel.com/issues/?jql=project%20%3D%20LU%20AND%20fixVer...
-If there are any issues not presently marked as blockers that you believe should be, please let me know
-We are trialling using the Fix In Release field for the 2.5 release to track work hoped to be included in the release; this means that a fuller picture of the progress on the release can be viewed at https://jira.hpdd.intel.com/browse/LU/fixforversion/10295#selectedTab=com...
-Feature Freeze is now in effect and 2.5 has entered its stabilization period. We encourage community members to test out the latest code on master and to use JIRA (https://jira.hpdd.intel.com) to report any bugs encountered
-You can also keep up to date with matters relating to the 2.5 release on the CDWG wiki - http://wiki.opensfs.org/Lustre_2.5.0
I have noticed when watching the metadata performance on a v2.4 MDS (llstat -i1 mdt) the rates drop to zero for seconds at a time sporadically. I never saw this on a v1.8.x server with comparable workloads. Looking at the debug logs when this happens it looks like it is stuck doing this:
00000004:00080000:4.0:1375970781.689383:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff8821062c4000, transno 8617232460
00000004:00080000:4.0:1375970781.689384:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff882d394d4c00, transno 8617232461
00000004:00080000:4.0:1375970781.689384:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883e03485800, transno 8617232462
00000004:00080000:4.0:1375970781.689385:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883479fcdc00, transno 8617232463
00000004:00080000:4.0:1375970781.689386:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883515c1f400, transno 8617232464
00000004:00080000:4.0:1375970781.689387:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883d76783000, transno 8617232465
I'm assuming this code was added in v2.x? Is it the expected behaviour that the rate of operations would "lock" until the syncs are completed?
/proc/fs/lustre/mds/MDS/mdt/stats @ 1375970781.677198
Name Cur.Count Cur.Rate #Events Unit last min avg max stddev
req_waittime 0 0 803871957 [usec] 0 2 6.63 10477 8.15
req_qdepth 0 0 803871957 [reqs] 0 0 0.00 12 0.07
req_active 0 0 803871957 [reqs] 0 1 2.67 16 1.79
req_timeout 0 0 803871957 [sec] 0 1 15.50 37 14.29
reqbuf_avail 0 0 1709293570[bufs] 0 47 63.79 64 0.59
ldlm_ibits_enqueue 0 0 499159709 [reqs] 0 1 1.00 1 0.00
mds_getattr 0 0 3152776 [usec] 0 8 346.15 1453985 6393.00
mds_getattr_lock 0 0 158194 [usec] 0 9 83.28 381776 1418.81
mds_connect 0 0 76 [usec] 0 14 3477.13 147911 21249.32
mds_disconnect 0 0 19 [usec] 0 26 1178.21 15598 3757.50
mds_getstatus 0 0 4 [usec] 0 9 13.75 16 3.20
mds_statfs 0 0 2904 [usec] 0 5 19.87 2115 67.53
mds_sync 0 0 3 [usec] 0 116 10933.67 32562 18730.69
mds_getxattr 0 0 50841 [usec] 0 6 12.89 92 3.91
obd_ping 0 0 530449 [usec] 0 3 11.87 3851 8.76
I am investigating why this workload seems to run much slower on v2.4 than v1.8.9. The workload is extremely hard link/unlink heavy as whole server filesystems are being backed up using "rsync --link-dest". Perhaps the unlink performance is slower as the MDS does it instead of the clients?
I am little bit puzzled my Reading is faster than Writing, should not be
other way around? Writing on single OST is about 400MiB/s using Lustre.
If I write directly to raid it brings 1200MiB/s and random 800MiB/s
Could you please advise where to dig or tuneup?
Thanks in advance
My recent IOR test shows:
mpirun -np 4 src/ior -a POSIX -r -w -b 32g -t 1m -o
/lustre/arm2arm/IORFile -O lustreStripeCount=-1 -v -i 3
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test#
#Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize
aggsize API RefNum
write 343.27 177.23 287.06 77.67 502.04247 0 4 4 3 0
0 1 0 0 1 34359738368 1048576 137438953472 POSIX 0
read 788.68 768.17 776.53 8.79 168.81345 0 4 4 3 0
0 1 0 0 1 34359738368 1048576 137438953472 POSIX 0
the mds-survey shows good performance:
thrhi=16 dir_count=16 file_count=200000 mds-survey
Thu Aug 1 12:05:12 CEST 2013 /usr/bin/mds-survey from araid062
mdt 1 file 200000 dir 16 thr 16 create 33506.27 [25998.91,36998.41]
lookup 790273.32 [790273.32,790273.32] md_getattr 474972.51
[474972.51,474972.51] setxattr 11483.39 [ 0.00,21999.03] destroy 73063.39
My hardware is following:
Raid box has a 24 SATA Disks(7200rpm with adaptec 6805controller raid6)
,DDR IB, 32GB Ram, 8core CPU.
Each box has a sigle port DDR IBs.
oss1, oss2- 34T - used 75%
mdt: 92G - used 3%
Software SL6.4, Lustre 2.4
Current active profile: latency-performance
Service tuned: enabled, running
Service ktune: enabled, running
I am looking for some testing, diagnostic and debugging tools for Lustre 2.4. I have 3 node setup up and running where one node is MGS/MDS, 1 node is OSS and another one is client. I am able to make the file system and mount the file systems on MGS and OSS and also able to mount MGS on Client. The only way I know this is running successfully as I do not receive any error.
Can someone suggest me tools or methods for Test the setup, diagnose the setup and debug the configuration?
EMC Data Storage Systems, Bangalore, India.