Re: [HPDD-discuss] [Lustre-devel] How should a LND handle a LNet get request?
by Tobias Groschup
Thanks for all your answers, they were really helpful. I think, I
understood how the communication for a LNetGet should work. At least, a
LNetGet does not cause LNet to crash any more.
But, the selftest does not work, it stops with an unknown RPC error. As
I said before, the LNetPut messages work without problems.
The selftest works this far: set up a session, add two nodes as a group
to the session, and add a test batch. Then, when a test is added to the
batch, the first LNetGet goes over the wire and the RPC error occurs.
I think, I copy the wrong memory ranges, but I could be wrong. So, maybe
someone here can help me.
Our network can do RDMA, the kernel API for the network accepts physical
addresses to describe where it should start reading and writing to the
memory.
On the initiator lnd_send is called with an lnet message with only a
memory descriptor attached, so the LND checks if this points to a iov or
kiov and attaches the iov.iov_base or maps the kiov page, and attaches
the physical address of the page (And cheks for offsets etc. to be
handled correctly).
On the target, lnd_recv is called with an kiov attached. So the LND
copies the data from the address of the mapped kiov page on the target
to the address it got from the initiator.
After the copy is done, both nodes get a notification from the network
device, and the LND calls lnet_finalize for the lnet messages on both nodes.
That's what I got from reading the o2iblnd code. Did I miss anything? At
the moment, I think that the LND reads or writes from/to the wrong
address. But I don't see where I go wrong, so maybe someone is able to
tell me where I mess up.
Thanks again for your help and kind regards
Tobias Groschup
8 years, 4 months
Follow up:The ost_connect operation failed with -30
by Hallstein Lohre
Hi,
Thank you Philippe,
Indeed there is something wrong here. The system is up and running, but
the same errors occur when I try to remount on ordinary 2.1.4 clients.
Seems there are two offending ost's making trouble, although they seems
ok. I'll check the hardware first.
Regards,
Hallstein
Le 15/01/2014 13:34, Hallstein Lohre a écrit :
>Hi,
>
>I am having problems installing a new system with Lustre 2.1.6 client
>talking to Lustre 2.1.4 servers.
>
>Dmesg on the client says:
>
>LustreError: 11-0: an error occurred while communicating with
>192.168.2.5@o2ib
><mailto:192.168.2.5@o2ib>. The ost_connect operation failed with -30
-30 /* Read-only file system */
-- Weill Philippe - Administrateur Systeme et Reseaux
CNRS/UPMC/IPSL LATMOS (UMR 8190)
Tour 45/46 3e Etage B302 - 4 Place Jussieu - 75252 Paris Cedex 05 -
FRANCE
Email:philippe.weill@latmos.ipsl.fr | tel:+33 0144274759 Fax:+33
0144273776
8 years, 4 months
The ost_connect operation failed with -30
by Hallstein Lohre
Hi,
I am having problems installing a new system with Lustre 2.1.6 client
talking to Lustre 2.1.4 servers.
Dmesg on the client says:
LustreError: 11-0: an error occurred while communicating with
192.168.2.5@o2ib. The ost_connect operation failed with -30
LustreError: 11-0: an error occurred while communicating with
192.168.2.3@o2ib. The ost_connect operation failed with -30
It sees that two OST's are involved, one on "192.168.2.3" and one on
"192.168.2.5". There are a total of 16 OST's in the setup.
Dmesg on the MDS says:
……….
LustreError: 10713:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x3582 sub-object on OST idx 9/1: rc = -30
LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x46fd sub-object on OST idx 9/1: rc = -30
LustreError: 24530:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x49a2 sub-object on OST idx 9/1: rc = -30
LustreError: 21647:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x5bb7 sub-object on OST idx 9/1: rc = -30
LustreError: 21647:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x5ddc sub-object on OST idx 9/1: rc = -30
LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x6c0b sub-object on OST idx 9/1: rc = -30
LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x7f87 sub-object on OST idx 9/1: rc = -30
LustreError: 24532:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x8237 sub-object on OST idx 9/1: rc = -30
LustreError: 24528:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x981b sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0xaddd sub-object on OST idx 9/1: rc = -30
LustreError: 24532:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0xc419 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0xc8db sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0xd4cc sub-object on OST idx 9/1: rc = -30
LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0xd720 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0xe7b3 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0xea01 sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0xfd5f sub-object on OST idx 9/1: rc = -30
LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x10cb1 sub-object on OST idx 9/1: rc = -30
LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x10f16 sub-object on OST idx 21/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x10f17 sub-object on OST idx 9/1: rc = -30
LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x124fe sub-object on OST idx 9/1: rc = -30
LustreError: 10713:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x1273a sub-object on OST idx 9/1: rc = -30
LustreError: 24528:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x1298b sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x1376f sub-object on OST idx 9/1: rc = -30
LustreError: 17092:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x139e0 sub-object on OST idx 9/1: rc = -30
LustreError: 27480:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x13c3e sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x15474 sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x15681 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x169b5 sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x16bf4 sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x16e4f sub-object on OST idx 9/1: rc = -30
LustreError: 31484:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x17f08 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error
creating fid 0x1814e sub-object on OST idx 9/1: rc = -30
……………..
Dmesg on one of the OSS’es says:
……………..
Lustre: 2053:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48
previous similar messages
LustreError: 2036:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error
starting handle for op 8 (71 credits): rc -30
LustreError: 2036:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start())
Skipped 51 previous similar messages
LustreError: 2040:0:(filter.c:383:filter_client_add()) unable to start
transaction: rc -30
LustreError: 2040:0:(filter.c:383:filter_client_add()) Skipped 48
previous similar messages
LustreError: 2040:0:(filter.c:407:filter_client_add()) error writing
last_rcvd client idx 32: rc -30
LustreError: 2040:0:(filter.c:407:filter_client_add()) Skipped 47
previous similar messages
Lustre: 2091:0:(ldlm_lib.c:952:target_handle_connect())
mtlustre-OST0009: connection from
24a25fe2-bb1f-f864-780f-842270fddd12@192.168.1.102@o2ib t0 exp (null)
cur 1389788167 last 0
Lustre: 2091:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48
previous similar messages
LustreError: 2142:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error
starting handle for op 8 (71 credits): rc -30
LustreError: 2142:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start())
Skipped 49 previous similar messages
LustreError: 2106:0:(filter.c:383:filter_client_add()) unable to start
transaction: rc -30
LustreError: 2106:0:(filter.c:383:filter_client_add()) Skipped 48
previous similar messages
LustreError: 2106:0:(filter.c:407:filter_client_add()) error writing
last_rcvd client idx 32: rc -30
LustreError: 2106:0:(filter.c:407:filter_client_add()) Skipped 48
previous similar messages
Lustre: 2028:0:(ldlm_lib.c:952:target_handle_connect())
mtlustre-OST0009: connection from
930bfd88-c41e-c38c-567f-6e326fb7680a@192.168.0.16@o2ib t0 exp (null) cur
1389788779 last 0
Lustre: 2028:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48
previous similar messages
LustreError: 2106:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error
starting handle for op 8 (71 credits): rc -30
LustreError: 2106:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start())
Skipped 48 previous similar messages
………………
Here is some more info:
Lustre server:
# rpm -qa | grep lustre
kernel-2.6.32-279.14.1.el6_lustre.x86_64
lustre-tests-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64
kernel-firmware-2.6.32-279.14.1.el6_lustre.x86_64
lustre-ldiskfs-3.3.0-2.6.32_279.14.1.el6_lustre.x86_64.x86_64
lustre-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64
lustre-modules-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64
Lustre client:
# rpm -qa | grep lustre
lustre-client-modules-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64
lustre-client-tests-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64
lustre-client-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64
[root@ansjos ~]# uname -a
Linux xxxx.yyy.zz 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed Jun 12 03:34:52
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
On the MDS:
lctl dl
0 UP mgc MGC192.168.3.1@o2ib eb992c1c-b363-a877-ab65-88c804d08706 5
1 UP lov mtlustre-clilov-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 4
2 UP lmv mtlustre-clilmv-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 4
3 UP mdc mtlustre-MDT0000-mdc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
4 UP osc mtlustre-OST0000-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
5 UP osc mtlustre-OST0001-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
6 UP osc mtlustre-OST0002-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
7 UP osc mtlustre-OST0003-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
8 UP osc mtlustre-OST0004-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
9 UP osc mtlustre-OST0005-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
10 UP osc mtlustre-OST0006-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
11 UP osc mtlustre-OST0007-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
12 UP osc mtlustre-OST0008-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
13 UP osc mtlustre-OST0009-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
14 UP osc mtlustre-OST000a-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
15 UP osc mtlustre-OST000b-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
16 UP osc mtlustre-OST000c-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
17 UP osc mtlustre-OST000d-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
18 UP osc mtlustre-OST000e-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
19 UP osc mtlustre-OST000f-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
20 UP osc mtlustre-OST0010-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
21 UP osc mtlustre-OST0011-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
22 UP osc mtlustre-OST0012-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
23 UP osc mtlustre-OST0013-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
24 UP osc mtlustre-OST0014-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
25 UP osc mtlustre-OST0015-osc-ffff88018fcfac00
24a25fe2-bb1f-f864-780f-842270fddd12 5
On "192.168.2.3":
lctl dl
0 UP mgc MGC192.168.3.1@o2ib fe9a97e3-5101-d1cb-95a6-6e304ba4f4e3 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter mtlustre-OST0007 mtlustre-OST0007_UUID 75
3 UP obdfilter mtlustre-OST0011 mtlustre-OST0011_UUID 75
4 UP obdfilter mtlustre-OST0008 mtlustre-OST0008_UUID 75
5 UP obdfilter mtlustre-OST0009 mtlustre-OST0009_UUID 71
On "192.168.2.5":
lctl dl
0 UP mgc MGC192.168.3.1@o2ib e6602327-9396-962d-3c82-5e2770358050 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter mtlustre-OST000d mtlustre-OST000d_UUID 75
3 UP obdfilter mtlustre-OST0015 mtlustre-OST0015_UUID 71
4 UP obdfilter mtlustre-OST0013 mtlustre-OST0013_UUID 75
5 UP obdfilter mtlustre-OST000e mtlustre-OST000e_UUID 75
6 UP obdfilter mtlustre-OST000f mtlustre-OST000f_UUID 75
Any clues? What to look for?
Best regards,
Hallstein Lohre
Alpha System AS
8 years, 4 months
lfs_migrate on an OST results in zero files inspite of being 60% used
by Kumar, Amit
Dear All,
I have yet another interesting thing happening and not able to understand.
One of the OST's that I want to migrate the data off of is 58% full and when I run the migrate script on the entire lustre file system, it only found few files migrated only couple of Gigs and now it does not find any files to migrate. Any idea how to interpret this? In the past when I was migrating an OST data it took a long long time because there literally millions of file to go through. Same is the situation here but every run of migration script just finished without moving anything anymore.
Any insight into this is greatly appreciated.
Best Regards,
Amit H. Kumar
8 years, 4 months
How should a LND handle a LNet get request?
by Tobias Groschup
Hello,
the LND which I am implementing is now capable of handling LNET_MSG_PUT.
Or, at least, the LNet self test can create a group of nodes.
Now it is time for a real ping test, which requires LNET_MSG_GET messages.
It is not clear to me, how to handle that kind of LNet messages. The
LNDs send function get called with to send a get message, but without
any attached data (in this message, kiov and iov and the number of iov
is 0). So, the LND send this messages header to the node from where the
data should be fetched. On this node, the header is passed to LNet for
parsing. LNet calls the receive LNDs function with a LNET_MSG_REPLY,
this time with data attached.
Now, I am not sure what to do with that reply message. Should it be send
back to the requesting node, in order to get information where the data
should be placed? Or is this information already in the get request?
I tried to find documentation about this, or tried to understand the how
this is done by other LNDs (including the loopback device), but it was
not very clear (to me).
Thanks for your help and kind regards
Tobias Groschup
8 years, 4 months
Draining an OST
by Bob Ball
We are running lustre 2.1.6 on SL6.4 systems. Most OST date back to
lustre 1.8.4 under SL5.x.
I now find it necessary to drain and reformat the underlying RAID volume
of one of these OST. I have done this several times in the past, under
lustre 1.8.4, and was highly satisfied with the outcome. However, I
find this somewhat more problematic under 2.1.6 now. Basically, in the
two examples so far, corrupted files have resulted.
I have used lfs_migrate to first drain, then refill the OST after it is
reformatted. It is much faster now than under 1.8.4, which is nice. Do
I have to do this on an idle file system though to avoid the
corruption? The two previous examples were still live, so it was
possible that the corrupted files were being accessed at the time?
Could this have been the cause of the problems?
What am I missing in doing this now under 2.1.6?
Thanks in advance,
bob
8 years, 4 months
Nominations requested for your 2014 OpenSFS Community Representative Director - Due January 31st, 2014
by OpenSFS Administration
Dear OpenSFS Adopters and Supporters,
Nominations are now being accepted for candidates to serve as the 2014
OpenSFS Community Representative Director on the OpenSFS Board of Directors.
This position will be open from April 2014 to April 2015.
Nominations must be emailed to admin(a)opensfs.org before 5pm Pacific Time,
Friday, January 31, 2013.
Please take a moment to give this nomination serious consideration. The
person selected for this position is responsible for representing the
OpenSFS Supporter and Adopter Participants and has a full Board seat and
voting rights to carry out that responsibility. Community participation is
growing and we want to be sure all Supporters and Adopters have a strong
voice in the decisions made by the OpenSFS Board. Your Community Board
Member is guided by Supporter and Adopter values as the Board makes
decisions for OpenSFS including how OpenSFS Development and Support funding
is spent.
We'd like to thank Tommy Minyard from Texas Advanced Computing Center for
his excellent service as the 2013 Community Representative Director.
Full details regarding the nomination process can be found online:
www.opensfs.org/opensfs-community-rep-election/
Sincerely,
Galen M. Shipman - Chairman, OpenSFS
8 years, 4 months
Lustre Transparency and Momentum for 2014
by OpenSFS Administration
Happy New Year everyone!
For our first 2014 post, we wanted to take a quick look back on the
accomplishments of last year. In 2013, OpenSFS added two new Participants --
Fujitsu and NCSA -- and Intel moved up to Promoter level and joined the
OpenSFS board. OpenSFS also pushed enhancements to our communications and
marketing efforts, added more mature management infrastructure, continued
funding the Lustre tree contract and added more Lustre development funding,
supported enhanced feature availability, and had a significant hand in
building a stronger foundation that the Lustre vendor ecosystem relies on
for their product lines. In particular, we continued to contribute
financially to Lustre. We spent over $3 million on Lustre technical
development and maintenance in 2013, our largest annual commitment to date.
Want to hear more? www.opensfs.org/lustre-transparency-momentum-2014/
Looking forward to working with you in 2014!
Sincerely,
The whole OpenSFS organization
8 years, 4 months
Invitation to Join LUG Planning Committee
by OpenSFS Administration
Hello OpenSFS participants,
Do you have opinions and suggestions for this year's LUG event? If so, we'd
love your input!
OpenSFS is forming the LUG 2014 planning committee and we're calling for
participants! The LUG planning committee is a vital group of volunteers
responsible for providing input, vetting presentation submissions, and
advising on event details. If you've attended LUG and have feedback on what
went well, suggested improvements, and speaking recommendations, we want to
hear from you!
On a company level, participation in the LUG planning committee will provide
insight to LUG content and sessions, allow your company to influence the
direction of LUG, and increase involvement in the Lustre community.
The LUG planning committee will meet 2-4 times a month in order to review
event plans, select presentation content, and ensure this year's LUG is a
smashing success. We look forward to participation from the broader Lustre
community.
If you're interested in participating in the LUG planning committee, please
contact OpenSFS Administration with the following information by Monday,
January 13th.
* Name
* Company
* Email
* Time zone/physical location
As a reminder, this year's LUG will be held in Miami, Florida, April 8-10th.
Sponsorship opportunities <http://www.opensfs.org/lug-2014-sponsorship/>
and the call for presentations
<http://www.opensfs.org/lug-2014-call-papers/> are now open! Stay tuned for
registration information!
Best Regards,
OpenSFS Administration
admin(a)opensfs.org
Ph: 503-619-0561
Open Scalable File Systems, Inc. is a strong and growing nonprofit
organization dedicated to the success of the LustreR file system. OpenSFS
was founded in 2010 to advance Lustre development, ensuring it remains
vendor-neutral, open, and free <http://lustre.opensfs.org/download-lustre/>
. Since its inception, OpenSFS has been responsible for advancing the Lustre
file system and delivering new releases
<http://lustre.opensfs.org/community-lustre-roadmap/> on behalf of the open
source community. Through working groups, events, and ongoing funding
initiatives, OpenSFS harnesses the power of collaborative development to
fuel innovation and growth of the Lustre file system worldwide.
8 years, 4 months
Changelog, OSD and Lustre 2.1/2.4
by DEGREMONT Aurelien
Hello
llog interface was redesigned and rebased on top of OSD between Lustre
2.1 and 2.4.
This is impacting Changelogs.
Now, in 2.4, changelog_catalog and changelog_users file are in root MDT
directory. In 2.1, there were in CONFIGS/ MDT directory.
Also, the changelog_catalogs were pointing to other files in OBJECTS/
directory. This directory does not exist anymore in Lustre 2.4
I'm not sure were those files are now, maybe in "0". There, filename
also have changed.
Am I correctly understanding the changes made in this area?
Was the file relocation something done in purpose?
Anyway, the upgrade case is not handle as a freshly updated MDT will
"forget" all the previous changelog users and changelog records. If this
is not know, I will open a ticket for that.
Aurélien
8 years, 4 months
