Example pacemaker files
by Roger Spellman
Hi,
The current Lustre manual recommends using Pacemaker with Lustre. However, it does not provide detailed examples. It provides a link to a wiki site, http://wiki.lustre.org/index.php/Using_Pacemaker_with_Lustre, which describes the process, but still does not give any specific examples.
Does anyone have config files for Pacemaker and Corosync that they'd be willing to share?
Thanks.
-Roger
7 years, 7 months
Redo for recoverable error ?
by Kumar, Amit
Dear All,
I am seeing a ton of these recently and based on reading this post http://lists.lustre.org/pipermail/lustre-discuss/2011-February/015065.html
It seems it has to do with mmap IO. Although I do not know if any of our user/application is performing mmap IO, my guess is mostly like they are.
Current version of our Lustre on MDS & OSS is: 1.8.5 and on Clients is 1.8.7;
Does these error indicate any kind of problem that needs to be addressed?
Please advise.
ON CLINET I SEE THE FOLLOWING,
CORRESPONDING SERVER MESSAGES AT THE END
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 9 previous similar messages
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 8 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067dd00800 x1439211562817922/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376338917 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 8 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 1 previous similar message
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6400 x1439211562999631/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339732 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 4 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13c00 x1439211562999918/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339747 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 4 previous similar messages
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 1 previous similar message
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067dd00800 x1439211563037211/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339925 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 1 previous similar message
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 8 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6000 x1439211563038445/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339970 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 8 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 1 previous similar message
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13400 x1439211563041360/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376340060 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 8438:0:(rw.c:198:ll_file_punch()) obd_truncate fails (-30) ino 29050451
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 10 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6400 x1439211691656856/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476163 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 9 previous similar messages
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 5 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6c00 x1439211691662173/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476184 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 5 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 6 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13800 x1439211691697906/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476223 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 5 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 15 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13400 x1439211691832310/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476292 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 14 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 30 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6400 x1439211691998133/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476438 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 27 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 54 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067eb13c00 x1439211692361454/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476739 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 49 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 1 previous similar message
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 2 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 4 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81067fdf6c00 x1439211693129332/t0 o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376477339 ref 2 fl Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 100 previous similar messages
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write operation failed with -30
LustreError: Skipped 111 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries, returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 8 previous similar messages
SERVER MESSAGES:
LustreError: 10258:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8105f9e02000 x1442836047652704/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376496995 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10258:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28 previous similar messages
LustreError: 14875:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 14875:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 48 previous similar messages
LustreError: 14875:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 14875:0:(filter.c:427:filter_client_add()) Skipped 43 previous similar messages
LustreError: 14875:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19177: rc -30
LustreError: 14875:0:(filter.c:451:filter_client_add()) Skipped 43 previous similar messages
LustreError: 14875:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8101551bb800 x1442836047654898/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376497598 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 14875:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 43 previous similar messages
LustreError: 10290:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10290:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous similar messages
LustreError: 10290:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10290:0:(filter.c:427:filter_client_add()) Skipped 26 previous similar messages
LustreError: 10290:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19204: rc -30
LustreError: 10290:0:(filter.c:451:filter_client_add()) Skipped 26 previous similar messages
LustreError: 10290:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81058f257000 x1442836047657674/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376498207 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10290:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26 previous similar messages
LustreError: 14915:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 14915:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 48 previous similar messages
LustreError: 14915:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 14915:0:(filter.c:427:filter_client_add()) Skipped 41 previous similar messages
LustreError: 14915:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19246: rc -30
LustreError: 14915:0:(filter.c:451:filter_client_add()) Skipped 41 previous similar messages
LustreError: 14915:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8101f2b6e000 x1443293932604564/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376498812 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 14915:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 41 previous similar messages
LustreError: 10123:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10123:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous similar messages
LustreError: 10123:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10123:0:(filter.c:427:filter_client_add()) Skipped 25 previous similar messages
LustreError: 10123:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19272: rc -30
LustreError: 10123:0:(filter.c:451:filter_client_add()) Skipped 25 previous similar messages
LustreError: 10123:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8104154f5c00 x1443293932605656/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376499437 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10123:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 25 previous similar messages
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 46 previous similar messages
LustreError: 10144:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10144:0:(filter.c:427:filter_client_add()) Skipped 40 previous similar messages
LustreError: 10144:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19313: rc -30
LustreError: 10144:0:(filter.c:451:filter_client_add()) Skipped 40 previous similar messages
LustreError: 10144:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81032ae86c00 x1443293932607309/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376500052 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10144:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 40 previous similar messages
LustreError: 14900:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 14900:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 37 previous similar messages
LustreError: 14900:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 14900:0:(filter.c:427:filter_client_add()) Skipped 32 previous similar messages
LustreError: 14900:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19346: rc -30
LustreError: 14900:0:(filter.c:451:filter_client_add()) Skipped 32 previous similar messages
LustreError: 14900:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81027b87bc00 x1443293932619476/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376500668 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 14900:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32 previous similar messages
LustreError: 10180:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10180:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 53 previous similar messages
LustreError: 10180:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10180:0:(filter.c:427:filter_client_add()) Skipped 46 previous similar messages
LustreError: 10180:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19393: rc -30
LustreError: 10180:0:(filter.c:451:filter_client_add()) Skipped 46 previous similar messages
LustreError: 10180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81061f6db800 x1442836047673405/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376501270 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10180:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 46 previous similar messages
LustreError: 10076:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10076:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 32 previous similar messages
LustreError: 10076:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10076:0:(filter.c:427:filter_client_add()) Skipped 27 previous similar messages
LustreError: 10076:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19421: rc -30
LustreError: 10076:0:(filter.c:451:filter_client_add()) Skipped 27 previous similar messages
LustreError: 10076:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81010e007400 x1442836047675443/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376501870 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10076:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27 previous similar messages
LustreError: 10367:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10367:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 55 previous similar messages
LustreError: 10367:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10367:0:(filter.c:427:filter_client_add()) Skipped 43 previous similar messages
LustreError: 10367:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19465: rc -30
LustreError: 10367:0:(filter.c:451:filter_client_add()) Skipped 43 previous similar messages
LustreError: 10367:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81009ff42800 x1442836047677670/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376502470 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10367:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 43 previous similar messages
LustreError: 10084:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10084:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 36 previous similar messages
LustreError: 10084:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10084:0:(filter.c:427:filter_client_add()) Skipped 28 previous similar messages
LustreError: 10084:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19494: rc -30
LustreError: 10084:0:(filter.c:451:filter_client_add()) Skipped 28 previous similar messages
LustreError: 10084:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810170e92850 x1443293932753685/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376503079 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10084:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28 previous similar messages
LustreError: 10397:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10397:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous similar messages
LustreError: 10397:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10397:0:(filter.c:427:filter_client_add()) Skipped 39 previous similar messages
LustreError: 10397:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19534: rc -30
LustreError: 10397:0:(filter.c:451:filter_client_add()) Skipped 39 previous similar messages
LustreError: 10397:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81029cedfc00 x1443293932886002/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376503687 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10397:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 39 previous similar messages
LustreError: 14958:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 14958:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 39 previous similar messages
LustreError: 14958:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 14958:0:(filter.c:427:filter_client_add()) Skipped 31 previous similar messages
LustreError: 14958:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19566: rc -30
LustreError: 14958:0:(filter.c:451:filter_client_add()) Skipped 31 previous similar messages
LustreError: 14958:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810611441000 x1443293933016614/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376504294 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 14958:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31 previous similar messages
LustreError: 10475:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10475:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 37 previous similar messages
LustreError: 10475:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10475:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar messages
LustreError: 10475:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19602: rc -30
LustreError: 10475:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar messages
LustreError: 10475:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810031d5dc00 x1442836047685269/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376504895 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10475:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages
LustreError: 10347:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10347:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 41 previous similar messages
LustreError: 10347:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10347:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar messages
LustreError: 10347:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19638: rc -30
LustreError: 10347:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar messages
LustreError: 10347:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810574a2b400 x1442836047687442/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376505520 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10347:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages
LustreError: 10165:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10165:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous similar messages
LustreError: 10165:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10165:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar messages
LustreError: 10165:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19674: rc -30
LustreError: 10165:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar messages
LustreError: 10165:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8104380fd000 x1443293933392255/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376506144 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10165:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages
LustreError: 10350:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10350:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 50 previous similar messages
LustreError: 10350:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10350:0:(filter.c:427:filter_client_add()) Skipped 38 previous similar messages
LustreError: 10350:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19713: rc -30
LustreError: 10350:0:(filter.c:451:filter_client_add()) Skipped 38 previous similar messages
LustreError: 10350:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8102593f6c00 x1442836047692767/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376506752 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10350:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 38 previous similar messages
LustreError: 10370:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10370:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 46 previous similar messages
LustreError: 10370:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10370:0:(filter.c:427:filter_client_add()) Skipped 36 previous similar messages
LustreError: 10370:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19750: rc -30
LustreError: 10370:0:(filter.c:451:filter_client_add()) Skipped 36 previous similar messages
LustreError: 10370:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff810527481400 x1442836047694671/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376507370 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10370:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 36 previous similar messages
LustreError: 10330:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10330:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 42 previous similar messages
LustreError: 10330:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10330:0:(filter.c:427:filter_client_add()) Skipped 37 previous similar messages
LustreError: 10330:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19788: rc -30
LustreError: 10330:0:(filter.c:451:filter_client_add()) Skipped 37 previous similar messages
LustreError: 10330:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff81063a334c00 x1443293933800716/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376507987 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10330:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 37 previous similar messages
LustreError: 10153:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10153:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous similar messages
LustreError: 10153:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10153:0:(filter.c:427:filter_client_add()) Skipped 37 previous similar messages
LustreError: 10153:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19826: rc -30
LustreError: 10153:0:(filter.c:451:filter_client_add()) Skipped 37 previous similar messages
LustreError: 10153:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8102c7ebf800 x1442836047701333/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376508595 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10153:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 37 previous similar messages
LustreError: 10256:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10256:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous similar messages
LustreError: 10256:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10256:0:(filter.c:427:filter_client_add()) Skipped 31 previous similar messages
LustreError: 10256:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19858: rc -30
LustreError: 10256:0:(filter.c:451:filter_client_add()) Skipped 31 previous similar messages
LustreError: 10256:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8101f41db800 x1442836047703557/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376509195 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10256:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31 previous similar messages
LustreError: 10170:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 10170:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous similar messages
LustreError: 10170:0:(filter.c:427:filter_client_add()) unable to start transaction: rc -30
LustreError: 10170:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar messages
LustreError: 10170:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx 19894: rc -30
LustreError: 10170:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar messages
LustreError: 10170:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30) req@ffff8101afc16c00 x1442836047705962/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1376509795 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10170:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar messages
Thank you,
Amit H. Kumar
7 years, 7 months
enabling file locking - lustre 1.8.8
by Kurt Strosahl
Hello,
Recently the question was asked if we could enable file locking on our lustre file system (because of automake/autoconf). We are currently running lustre 1.8.8 on our mds/loss servers, which apparently doesn't support (or it isn't enabled) file locking.
Is this something that could be done, non-destructively, to the current system... or would we have to upgrade to a newer version?
w/r,
Kurt J. Strosahl
System Administrator
Scientific Computing Group, Thomas Jefferson National Accelerator Facility
7 years, 7 months
(no subject)
by Raymond Lo
Hi all,
I have setup 1.8 on one MDT and 4 OSS
Recently the MDT crashed and have to recover from backup, then filesystem
is mountable but cannot write onto the filesystem (probably due to
inconsistency between MDT and OSS)
Pls, can anyone give some advice, thanks.
7 years, 8 months
v2.4 metadata pauses
by Daire Byrne
Hi,
I have noticed when watching the metadata performance on a v2.4 MDS (llstat -i1 mdt) the rates drop to zero for seconds at a time sporadically. I never saw this on a v1.8.x server with comparable workloads. Looking at the debug logs when this happens it looks like it is stuck doing this:
00000004:00080000:4.0:1375970781.689383:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff8821062c4000, transno 8617232460
00000004:00080000:4.0:1375970781.689384:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff882d394d4c00, transno 8617232461
00000004:00080000:4.0:1375970781.689384:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883e03485800, transno 8617232462
00000004:00080000:4.0:1375970781.689385:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883479fcdc00, transno 8617232463
00000004:00080000:4.0:1375970781.689386:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883515c1f400, transno 8617232464
00000004:00080000:4.0:1375970781.689387:0:4870:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883d76783000, transno 8617232465
I'm assuming this code was added in v2.x? Is it the expected behaviour that the rate of operations would "lock" until the syncs are completed?
/proc/fs/lustre/mds/MDS/mdt/stats @ 1375970781.677198
Name Cur.Count Cur.Rate #Events Unit last min avg max stddev
req_waittime 0 0 803871957 [usec] 0 2 6.63 10477 8.15
req_qdepth 0 0 803871957 [reqs] 0 0 0.00 12 0.07
req_active 0 0 803871957 [reqs] 0 1 2.67 16 1.79
req_timeout 0 0 803871957 [sec] 0 1 15.50 37 14.29
reqbuf_avail 0 0 1709293570[bufs] 0 47 63.79 64 0.59
ldlm_ibits_enqueue 0 0 499159709 [reqs] 0 1 1.00 1 0.00
mds_getattr 0 0 3152776 [usec] 0 8 346.15 1453985 6393.00
mds_getattr_lock 0 0 158194 [usec] 0 9 83.28 381776 1418.81
mds_connect 0 0 76 [usec] 0 14 3477.13 147911 21249.32
mds_disconnect 0 0 19 [usec] 0 26 1178.21 15598 3757.50
mds_getstatus 0 0 4 [usec] 0 9 13.75 16 3.20
mds_statfs 0 0 2904 [usec] 0 5 19.87 2115 67.53
mds_sync 0 0 3 [usec] 0 116 10933.67 32562 18730.69
mds_getxattr 0 0 50841 [usec] 0 6 12.89 92 3.91
obd_ping 0 0 530449 [usec] 0 3 11.87 3851 8.76
I am investigating why this workload seems to run much slower on v2.4 than v1.8.9. The workload is extremely hard link/unlink heavy as whole server filesystems are being backed up using "rsync --link-dest". Perhaps the unlink performance is slower as the MDS does it instead of the clients?
Regards,
Daire
7 years, 8 months
Why my lustre 2.4 is faster on Reading and slow on write?
by Arman Khalatyan
Hello,
I am little bit puzzled my Reading is faster than Writing, should not be
other way around? Writing on single OST is about 400MiB/s using Lustre.
If I write directly to raid it brings 1200MiB/s and random 800MiB/s
Could you please advise where to dig or tuneup?
Thanks in advance
Arman.
tuned-admin
My recent IOR test shows:
mpirun -np 4 src/ior -a POSIX -r -w -b 32g -t 1m -o
/lustre/arm2arm/IORFile -O lustreStripeCount=-1 -v -i 3
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test#
#Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize
aggsize API RefNum
write 343.27 177.23 287.06 77.67 502.04247 0 4 4 3 0
0 1 0 0 1 34359738368 1048576 137438953472 POSIX 0
read 788.68 768.17 776.53 8.79 168.81345 0 4 4 3 0
0 1 0 0 1 34359738368 1048576 137438953472 POSIX 0
the mds-survey shows good performance:
thrhi=16 dir_count=16 file_count=200000 mds-survey
Thu Aug 1 12:05:12 CEST 2013 /usr/bin/mds-survey from araid062
mdt 1 file 200000 dir 16 thr 16 create 33506.27 [25998.91,36998.41]
lookup 790273.32 [790273.32,790273.32] md_getattr 474972.51
[474972.51,474972.51] setxattr 11483.39 [ 0.00,21999.03] destroy 73063.39
[63998.59,63998.59]
done!
My hardware is following:
Raid box has a 24 SATA Disks(7200rpm with adaptec 6805controller raid6)
,DDR IB, 32GB Ram, 8core CPU.
RaidBox1: mds/mdt+oss2
drbd |
RaidBox2: mds/mdt(slave)+oss1
Each box has a sigle port DDR IBs.
oss1, oss2- 34T - used 75%
mdt: 92G - used 3%
Software SL6.4, Lustre 2.4
kernel 2.6.32-358.6.2
tuned-adm active
Current active profile: latency-performance
Service tuned: enabled, running
Service ktune: enabled, running
7 years, 8 months
Testing, Diagnostic and Debug tools for Lustre 2.4
by Singhal, Upanshu
Hello All,
I am looking for some testing, diagnostic and debugging tools for Lustre 2.4. I have 3 node setup up and running where one node is MGS/MDS, 1 node is OSS and another one is client. I am able to make the file system and mount the file systems on MGS and OSS and also able to mount MGS on Client. The only way I know this is running successfully as I do not receive any error.
Can someone suggest me tools or methods for Test the setup, diagnose the setup and debug the configuration?
Thanks,
-Upanshu
Upanshu Singhal
EMC Data Storage Systems, Bangalore, India.
Phone: 91-80-67375604
7 years, 8 months