August 2013 - HPDD-discuss

by Roger Spellman

Hi, The current Lustre manual recommends using Pacemaker with Lustre. However, it does not provide detailed examples. It provides a link to a wiki site, http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre, which describes the process, but still does not give any specific examples. Does anyone have config files for Pacemaker and Corosync that they'd be willing to share? Thanks. -Roger

7 years, 10 months

1
0
0 / 0

Lustre 2.5 update - August 16th 2013

by Jones, Peter A

Hi there Here is an update on the Lustre 2.5 release. Landings ======== -A number of landings made - see http://git.whamcloud.com/?p=fs/lustre-release.git;a=shortlog;h=refs/heads... -Further to the final HSM feature patches landed last week, the HSM test script has now landed (LU-3561) Testing ======= - Testing on the 2.4.53 tag is complete; testing on the 2.4.90 tag is ongoing (the bump in tag numbers is to indicate feature freeze has passed) Blockers ======== -Full list available at https://jira.hpdd.intel.com/issues/?jql=project%20%3D%20LU%20AND%20fixVer... -If there are any issues not presently marked as blockers that you believe should be, please let me know Other ===== -We are trialling using the Fix In Release field for the 2.5 release to track work hoped to be included in the release; this means that a fuller picture of the progress on the release can be viewed at https://jira.hpdd.intel.com/browse/LU/fixforversion/10295#selectedTab=com... -Feature Freeze is now in effect and 2.5 has entered its stabilization period. We encourage community members to test out the latest code on master and to use JIRA (https://jira.hpdd.intel.com<https://jira.hpdd.intel.com/>) to report any bugs encountered -You can also keep up to date with matters relating to the 2.5 release on the CDWG wiki - http://wiki.opensfs.org/Lustre_2.5.0 Thanks Peter

7 years, 10 months

1
0
0 / 0

Redo for recoverable error ?

by Kumar, Amit

Dear All, I am seeing a ton of these recently and based on reading this post http://lists.lustre.org/pipermail/lustre-discuss/2011-February/015065.html It seems it has to do with mmap IO. Although I do not know if any of our user/application is performing mmap IO, my guess is mostly like they are. Current version of our Lustre on MDS & OSS is: 1.8.5 and on Clients is 1.8.7; Does these error indicate any kind of problem that needs to be addressed? Please advise. ON CLINET I SEE THE FOLLOWING, CORRESPONDING SERVER MESSAGES AT THE END LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 9 previous similar messages LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 8 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067dd00800 x1439211562817922/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376338917 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 8 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 1 previous similar message LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6400 x1439211562999631/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339732 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 4 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13c00 x1439211562999918/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339747 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 4 previous similar messages LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 1 previous similar message LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067dd00800 x1439211563037211/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339925 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 1 previous similar message LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 8 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6000 x1439211563038445/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339970 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 8 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 1 previous similar message LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13400 x1439211563041360/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376340060 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 8438:0:(rw.c:198:ll_file_punch()) obd_truncate fails (-30) ino 29050451 LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 10 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6400 x1439211691656856/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476163 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 9 previous similar messages LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 5 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6c00 x1439211691662173/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476184 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 5 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 6 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13800 x1439211691697906/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476223 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 5 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 15 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13400 x1439211691832310/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476292 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 14 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 30 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6400 x1439211691998133/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476438 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 27 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 54 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13c00 x1439211692361454/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476739 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 49 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 1 previous similar message LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 2 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 4 previous similar messages LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6c00 x1439211693129332/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376477339 ref 2 fl Interpret:R/0/0 rc -30/-30 LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 100 previous similar messages LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30 LustreError: Skipped 111 previous similar messages LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 8 previous similar messages SERVER MESSAGES: LustreError: 10258:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8105f9e02000 x1442836047652704/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376496995 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10258:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28 previous similar messages LustreError: 14875:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 14875:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 48 previous similar messages LustreError: 14875:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 14875:0:(filter.c:427:filter_client_add()) Skipped 43 previous similar messages LustreError: 14875:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19177: rc -30 LustreError: 14875:0:(filter.c:451:filter_client_add()) Skipped 43 previous similar messages LustreError: 14875:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8101551bb800 x1442836047654898/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376497598 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 14875:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 43 previous similar messages LustreError: 10290:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10290:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous similar messages LustreError: 10290:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10290:0:(filter.c:427:filter_client_add()) Skipped 26 previous similar messages LustreError: 10290:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19204: rc -30 LustreError: 10290:0:(filter.c:451:filter_client_add()) Skipped 26 previous similar messages LustreError: 10290:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81058f257000 x1442836047657674/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376498207 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10290:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26 previous similar messages LustreError: 14915:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 14915:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 48 previous similar messages LustreError: 14915:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 14915:0:(filter.c:427:filter_client_add()) Skipped 41 previous similar messages LustreError: 14915:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19246: rc -30 LustreError: 14915:0:(filter.c:451:filter_client_add()) Skipped 41 previous similar messages LustreError: 14915:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8101f2b6e000 x1443293932604564/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376498812 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 14915:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 41 previous similar messages LustreError: 10123:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10123:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous similar messages LustreError: 10123:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10123:0:(filter.c:427:filter_client_add()) Skipped 25 previous similar messages LustreError: 10123:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19272: rc -30 LustreError: 10123:0:(filter.c:451:filter_client_add()) Skipped 25 previous similar messages LustreError: 10123:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8104154f5c00 x1443293932605656/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376499437 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10123:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 25 previous similar messages LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 46 previous similar messages LustreError: 10144:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10144:0:(filter.c:427:filter_client_add()) Skipped 40 previous similar messages LustreError: 10144:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19313: rc -30 LustreError: 10144:0:(filter.c:451:filter_client_add()) Skipped 40 previous similar messages LustreError: 10144:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81032ae86c00 x1443293932607309/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376500052 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10144:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 40 previous similar messages LustreError: 14900:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 14900:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 37 previous similar messages LustreError: 14900:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 14900:0:(filter.c:427:filter_client_add()) Skipped 32 previous similar messages LustreError: 14900:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19346: rc -30 LustreError: 14900:0:(filter.c:451:filter_client_add()) Skipped 32 previous similar messages LustreError: 14900:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81027b87bc00 x1443293932619476/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376500668 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 14900:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32 previous similar messages LustreError: 10180:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10180:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 53 previous similar messages LustreError: 10180:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10180:0:(filter.c:427:filter_client_add()) Skipped 46 previous similar messages LustreError: 10180:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19393: rc -30 LustreError: 10180:0:(filter.c:451:filter_client_add()) Skipped 46 previous similar messages LustreError: 10180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81061f6db800 x1442836047673405/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376501270 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10180:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 46 previous similar messages LustreError: 10076:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10076:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 32 previous similar messages LustreError: 10076:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10076:0:(filter.c:427:filter_client_add()) Skipped 27 previous similar messages LustreError: 10076:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19421: rc -30 LustreError: 10076:0:(filter.c:451:filter_client_add()) Skipped 27 previous similar messages LustreError: 10076:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81010e007400 x1442836047675443/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376501870 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10076:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27 previous similar messages LustreError: 10367:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10367:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 55 previous similar messages LustreError: 10367:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10367:0:(filter.c:427:filter_client_add()) Skipped 43 previous similar messages LustreError: 10367:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19465: rc -30 LustreError: 10367:0:(filter.c:451:filter_client_add()) Skipped 43 previous similar messages LustreError: 10367:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81009ff42800 x1442836047677670/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376502470 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10367:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 43 previous similar messages LustreError: 10084:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10084:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 36 previous similar messages LustreError: 10084:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10084:0:(filter.c:427:filter_client_add()) Skipped 28 previous similar messages LustreError: 10084:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19494: rc -30 LustreError: 10084:0:(filter.c:451:filter_client_add()) Skipped 28 previous similar messages LustreError: 10084:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810170e92850 x1443293932753685/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376503079 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10084:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28 previous similar messages LustreError: 10397:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10397:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous similar messages LustreError: 10397:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10397:0:(filter.c:427:filter_client_add()) Skipped 39 previous similar messages LustreError: 10397:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19534: rc -30 LustreError: 10397:0:(filter.c:451:filter_client_add()) Skipped 39 previous similar messages LustreError: 10397:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81029cedfc00 x1443293932886002/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376503687 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10397:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 39 previous similar messages LustreError: 14958:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 14958:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 39 previous similar messages LustreError: 14958:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 14958:0:(filter.c:427:filter_client_add()) Skipped 31 previous similar messages LustreError: 14958:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19566: rc -30 LustreError: 14958:0:(filter.c:451:filter_client_add()) Skipped 31 previous similar messages LustreError: 14958:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810611441000 x1443293933016614/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376504294 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 14958:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31 previous similar messages LustreError: 10475:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10475:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 37 previous similar messages LustreError: 10475:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10475:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar messages LustreError: 10475:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19602: rc -30 LustreError: 10475:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar messages LustreError: 10475:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810031d5dc00 x1442836047685269/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376504895 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10475:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages LustreError: 10347:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10347:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 41 previous similar messages LustreError: 10347:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10347:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar messages LustreError: 10347:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19638: rc -30 LustreError: 10347:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar messages LustreError: 10347:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810574a2b400 x1442836047687442/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376505520 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10347:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages LustreError: 10165:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10165:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous similar messages LustreError: 10165:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10165:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar messages LustreError: 10165:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19674: rc -30 LustreError: 10165:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar messages LustreError: 10165:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8104380fd000 x1443293933392255/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376506144 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10165:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages LustreError: 10350:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10350:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 50 previous similar messages LustreError: 10350:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10350:0:(filter.c:427:filter_client_add()) Skipped 38 previous similar messages LustreError: 10350:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19713: rc -30 LustreError: 10350:0:(filter.c:451:filter_client_add()) Skipped 38 previous similar messages LustreError: 10350:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8102593f6c00 x1442836047692767/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376506752 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10350:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 38 previous similar messages LustreError: 10370:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10370:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 46 previous similar messages LustreError: 10370:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10370:0:(filter.c:427:filter_client_add()) Skipped 36 previous similar messages LustreError: 10370:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19750: rc -30 LustreError: 10370:0:(filter.c:451:filter_client_add()) Skipped 36 previous similar messages LustreError: 10370:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810527481400 x1442836047694671/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376507370 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10370:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 36 previous similar messages LustreError: 10330:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10330:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 42 previous similar messages LustreError: 10330:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10330:0:(filter.c:427:filter_client_add()) Skipped 37 previous similar messages LustreError: 10330:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19788: rc -30 LustreError: 10330:0:(filter.c:451:filter_client_add()) Skipped 37 previous similar messages LustreError: 10330:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81063a334c00 x1443293933800716/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376507987 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10330:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 37 previous similar messages LustreError: 10153:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10153:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous similar messages LustreError: 10153:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10153:0:(filter.c:427:filter_client_add()) Skipped 37 previous similar messages LustreError: 10153:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19826: rc -30 LustreError: 10153:0:(filter.c:451:filter_client_add()) Skipped 37 previous similar messages LustreError: 10153:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8102c7ebf800 x1442836047701333/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376508595 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10153:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 37 previous similar messages LustreError: 10256:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10256:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous similar messages LustreError: 10256:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10256:0:(filter.c:427:filter_client_add()) Skipped 31 previous similar messages LustreError: 10256:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19858: rc -30 LustreError: 10256:0:(filter.c:451:filter_client_add()) Skipped 31 previous similar messages LustreError: 10256:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8101f41db800 x1442836047703557/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376509195 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10256:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31 previous similar messages LustreError: 10170:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 10170:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous similar messages LustreError: 10170:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30 LustreError: 10170:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar messages LustreError: 10170:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19894: rc -30 LustreError: 10170:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar messages LustreError: 10170:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8101afc16c00 x1442836047705962/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376509795 ref 1 fl Interpret:/0/0 rc -30/0 LustreError: 10170:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages Thank you, Amit H. Kumar

7 years, 10 months

2
7
0 / 0

enabling file locking - lustre 1.8.8

by Kurt Strosahl

Hello, Recently the question was asked if we could enable file locking on our lustre file system (because of automake/autoconf). We are currently running lustre 1.8.8 on our mds/loss servers, which apparently doesn't support (or it isn't enabled) file locking. Is this something that could be done, non-destructively, to the current system... or would we have to upgrade to a newer version? w/r, Kurt J. Strosahl System Administrator Scientific Computing Group, Thomas Jefferson National Accelerator Facility

7 years, 10 months

2
1
0 / 0

(no subject)

by Raymond Lo

Hi all, I have setup 1.8 on one MDT and 4 OSS Recently the MDT crashed and have to recover from backup, then filesystem is mountable but cannot write onto the filesystem (probably due to inconsistency between MDT and OSS) Pls, can anyone give some advice, thanks.

7 years, 10 months

1
0
0 / 0

Lustre 2.5 update - August 2nd 2013

by Jones, Peter A

Hi there Here is an update on the Lustre 2.5 release. Landings ======== -A number of landings made - see http://git.whamcloud.com/?p=fs/lustre-release.git;a=shortlog;h=refs/heads... -Now that the Feature Freeze has passed, the full list of features in 2.5 is known Xattr Cache (LU-2869) LNET router improvements (LU-2934/2950) Errno translation (step towards sparc support) (LU-2743) Permanent Parameters option for lctl set_param (LU-3155) Reduce ldlm_poold cpu utilization (LU-2924) HSM (LU-3608) There is still some remaining work for this to be feature complete. As per the discussion on the CDWG call on Wednesday we are going to see we have extended the landing window for these patches only in view of the high interest in this feature and the belief that this work is close to completion. I will send out another update early next week with the final outcome on this. * POSIX Copy tool http://review.whamcloud.com/#/c/4737/ * Disaster Recovery http://review.whamcloud.com/#/c/7027/ * Coordinator http://review.whamcloud.com/#/c/6912/ Testing ======= - Testing on the 2.4.52 tag is complete; testing on the 2.4.53 tag is ongoing Blockers ======== -Full list available at https://jira.hpdd.intel.com/issues/?jql=project%20%3D%20LU%20AND%20fixVer... -If there are any issues not presently marked as blockers that you believe should be, please let me know Other ===== -We are trialling using the Fix In Release field for the 2.5 release to track work hoped to be included in the release; this means that a fuller picture of the progress on the release can be viewed at https://jira.hpdd.intel.com/browse/LU/fixforversion/10295#selectedTab=com... -Feature Freeze is now in effect and 2.5 has entered its stabilization period. We encourage community members to test out the latest code on master and to use JIRA (https://jira.hpdd.intel.com) to report any bugs encountered -You can also keep up to date with matters relating to the 2.5 release on the CDWG wiki - http://wiki.opensfs.org/Lustre_2.5.0 Thanks Peter

7 years, 10 months

1
1
0 / 0

v2.4 metadata pauses

by Daire Byrne

Hi, I have noticed when watching the metadata performance on a v2.4 MDS (llstat -i1 mdt) the rates drop to zero for seconds at a time sporadically. I never saw this on a v1.8.x server with comparable workloads. Looking at the debug logs when this happens it looks like it is stuck doing this: 00000004:00080000:4.0:1375970781.689383:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff8821062c4000, transno 8617232460 00000004:00080000:4.0:1375970781.689384:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff882d394d4c00, transno 8617232461 00000004:00080000:4.0:1375970781.689384:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883e03485800, transno 8617232462 00000004:00080000:4.0:1375970781.689385:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883479fcdc00, transno 8617232463 00000004:00080000:4.0:1375970781.689386:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883515c1f400, transno 8617232464 00000004:00080000:4.0:1375970781.689387:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883d76783000, transno 8617232465 I'm assuming this code was added in v2.x? Is it the expected behaviour that the rate of operations would "lock" until the syncs are completed? /proc/fs/lustre/mds/MDS/mdt/stats @ 1375970781.677198 Name Cur.Count Cur.Rate #Events Unit last min avg max stddev req_waittime 0 0 803871957 [usec] 0 2 6.63 10477 8.15 req_qdepth 0 0 803871957 [reqs] 0 0 0.00 12 0.07 req_active 0 0 803871957 [reqs] 0 1 2.67 16 1.79 req_timeout 0 0 803871957 [sec] 0 1 15.50 37 14.29 reqbuf_avail 0 0 1709293570[bufs] 0 47 63.79 64 0.59 ldlm_ibits_enqueue 0 0 499159709 [reqs] 0 1 1.00 1 0.00 mds_getattr 0 0 3152776 [usec] 0 8 346.15 1453985 6393.00 mds_getattr_lock 0 0 158194 [usec] 0 9 83.28 381776 1418.81 mds_connect 0 0 76 [usec] 0 14 3477.13 147911 21249.32 mds_disconnect 0 0 19 [usec] 0 26 1178.21 15598 3757.50 mds_getstatus 0 0 4 [usec] 0 9 13.75 16 3.20 mds_statfs 0 0 2904 [usec] 0 5 19.87 2115 67.53 mds_sync 0 0 3 [usec] 0 116 10933.67 32562 18730.69 mds_getxattr 0 0 50841 [usec] 0 6 12.89 92 3.91 obd_ping 0 0 530449 [usec] 0 3 11.87 3851 8.76 I am investigating why this workload seems to run much slower on v2.4 than v1.8.9. The workload is extremely hard link/unlink heavy as whole server filesystems are being backed up using "rsync --link-dest". Perhaps the unlink performance is slower as the MDS does it instead of the clients? Regards, Daire

7 years, 10 months

2
2
0 / 0

Why my lustre 2.4 is faster on Reading and slow on write?

by Arman Khalatyan

Hello, I am little bit puzzled my Reading is faster than Writing, should not be other way around? Writing on single OST is about 400MiB/s using Lustre. If I write directly to raid it brings 1200MiB/s and random 800MiB/s Could you please advise where to dig or tuneup? Thanks in advance Arman. tuned-admin My recent IOR test shows: mpirun -np 4 src/ior -a POSIX -r -w -b 32g -t 1m -o /lustre/arm2arm/IORFile -O lustreStripeCount=-1 -v -i 3 Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 343.27 177.23 287.06 77.67 502.04247 0 4 4 3 0 0 1 0 0 1 34359738368 1048576 137438953472 POSIX 0 read 788.68 768.17 776.53 8.79 168.81345 0 4 4 3 0 0 1 0 0 1 34359738368 1048576 137438953472 POSIX 0 the mds-survey shows good performance: thrhi=16 dir_count=16 file_count=200000 mds-survey Thu Aug 1 12:05:12 CEST 2013 /usr/bin/mds-survey from araid062 mdt 1 file 200000 dir 16 thr 16 create 33506.27 [25998.91,36998.41] lookup 790273.32 [790273.32,790273.32] md_getattr 474972.51 [474972.51,474972.51] setxattr 11483.39 [ 0.00,21999.03] destroy 73063.39 [63998.59,63998.59] done! My hardware is following: Raid box has a 24 SATA Disks(7200rpm with adaptec 6805controller raid6) ,DDR IB, 32GB Ram, 8core CPU. RaidBox1: mds/mdt+oss2 drbd | RaidBox2: mds/mdt(slave)+oss1 Each box has a sigle port DDR IBs. oss1, oss2- 34T - used 75% mdt: 92G - used 3% Software SL6.4, Lustre 2.4 kernel 2.6.32-358.6.2 tuned-adm active Current active profile: latency-performance Service tuned: enabled, running Service ktune: enabled, running

7 years, 10 months

1
1
0 / 0

Unable to download the last succesfull build rpms from Jenkins.

by Arman Khalatyan

hello, I would like to install lustre with ZFS from build.whamcloud.com/job/lustre-master/arch=x86_64,build_type=server,distr... I am able to download all files except zfs and spl related ones. Should I get special account on jenkins? Is there reason why it is not accessible for public? Thanks, Arman.

7 years, 10 months

2
1
0 / 0

Testing, Diagnostic and Debug tools for Lustre 2.4

by Singhal, Upanshu

Hello All, I am looking for some testing, diagnostic and debugging tools for Lustre 2.4. I have 3 node setup up and running where one node is MGS/MDS, 1 node is OSS and another one is client. I am able to make the file system and mount the file systems on MGS and OSS and also able to mount MGS on Client. The only way I know this is running successfully as I do not receive any error. Can someone suggest me tools or methods for Test the setup, diagnose the setup and debug the configuration? Thanks, -Upanshu Upanshu Singhal EMC Data Storage Systems, Bangalore, India. Phone: 91-80-67375604

7 years, 10 months

1
0
0 / 0

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

HPDD-discuss August 2013