Thanks for all your answers, they were really helpful. I think, I
understood how the communication for a LNetGet should work. At least, a
LNetGet does not cause LNet to crash any more.
But, the selftest does not work, it stops with an unknown RPC error. As
I said before, the LNetPut messages work without problems.
The selftest works this far: set up a session, add two nodes as a group
to the session, and add a test batch. Then, when a test is added to the
batch, the first LNetGet goes over the wire and the RPC error occurs.
I think, I copy the wrong memory ranges, but I could be wrong. So, maybe
someone here can help me.
Our network can do RDMA, the kernel API for the network accepts physical
addresses to describe where it should start reading and writing to the
On the initiator lnd_send is called with an lnet message with only a
memory descriptor attached, so the LND checks if this points to a iov or
kiov and attaches the iov.iov_base or maps the kiov page, and attaches
the physical address of the page (And cheks for offsets etc. to be
On the target, lnd_recv is called with an kiov attached. So the LND
copies the data from the address of the mapped kiov page on the target
to the address it got from the initiator.
After the copy is done, both nodes get a notification from the network
device, and the LND calls lnet_finalize for the lnet messages on both nodes.
That's what I got from reading the o2iblnd code. Did I miss anything? At
the moment, I think that the LND reads or writes from/to the wrong
address. But I don't see where I go wrong, so maybe someone is able to
tell me where I mess up.
Thanks again for your help and kind regards
Thank you Philippe,
Indeed there is something wrong here. The system is up and running, but
the same errors occur when I try to remount on ordinary 2.1.4 clients.
Seems there are two offending ost's making trouble, although they seems
ok. I'll check the hardware first.
Le 15/01/2014 13:34, Hallstein Lohre a écrit :
>I am having problems installing a new system with Lustre 2.1.6 client
>talking to Lustre 2.1.4 servers.
>Dmesg on the client says:
>LustreError: 11-0: an error occurred while communicating with
><mailto:192.168.2.5@o2ib>. The ost_connect operation failed with -30
-30 /* Read-only file system */
-- Weill Philippe - Administrateur Systeme et Reseaux
CNRS/UPMC/IPSL LATMOS (UMR 8190)
Tour 45/46 3e Etage B302 - 4 Place Jussieu - 75252 Paris Cedex 05 -
Email:firstname.lastname@example.org | tel:+33 0144274759 Fax:+33
I have yet another interesting thing happening and not able to understand.
One of the OST's that I want to migrate the data off of is 58% full and when I run the migrate script on the entire lustre file system, it only found few files migrated only couple of Gigs and now it does not find any files to migrate. Any idea how to interpret this? In the past when I was migrating an OST data it took a long long time because there literally millions of file to go through. Same is the situation here but every run of migration script just finished without moving anything anymore.
Any insight into this is greatly appreciated.
Amit H. Kumar
the LND which I am implementing is now capable of handling LNET_MSG_PUT.
Or, at least, the LNet self test can create a group of nodes.
Now it is time for a real ping test, which requires LNET_MSG_GET messages.
It is not clear to me, how to handle that kind of LNet messages. The
LNDs send function get called with to send a get message, but without
any attached data (in this message, kiov and iov and the number of iov
is 0). So, the LND send this messages header to the node from where the
data should be fetched. On this node, the header is passed to LNet for
parsing. LNet calls the receive LNDs function with a LNET_MSG_REPLY,
this time with data attached.
Now, I am not sure what to do with that reply message. Should it be send
back to the requesting node, in order to get information where the data
should be placed? Or is this information already in the get request?
I tried to find documentation about this, or tried to understand the how
this is done by other LNDs (including the loopback device), but it was
not very clear (to me).
Thanks for your help and kind regards
We are running lustre 2.1.6 on SL6.4 systems. Most OST date back to
lustre 1.8.4 under SL5.x.
I now find it necessary to drain and reformat the underlying RAID volume
of one of these OST. I have done this several times in the past, under
lustre 1.8.4, and was highly satisfied with the outcome. However, I
find this somewhat more problematic under 2.1.6 now. Basically, in the
two examples so far, corrupted files have resulted.
I have used lfs_migrate to first drain, then refill the OST after it is
reformatted. It is much faster now than under 1.8.4, which is nice. Do
I have to do this on an idle file system though to avoid the
corruption? The two previous examples were still live, so it was
possible that the corrupted files were being accessed at the time?
Could this have been the cause of the problems?
What am I missing in doing this now under 2.1.6?
Thanks in advance,
Dear OpenSFS Adopters and Supporters,
Nominations are now being accepted for candidates to serve as the 2014
OpenSFS Community Representative Director on the OpenSFS Board of Directors.
This position will be open from April 2014 to April 2015.
Nominations must be emailed to admin(a)opensfs.org before 5pm Pacific Time,
Friday, January 31, 2013.
Please take a moment to give this nomination serious consideration. The
person selected for this position is responsible for representing the
OpenSFS Supporter and Adopter Participants and has a full Board seat and
voting rights to carry out that responsibility. Community participation is
growing and we want to be sure all Supporters and Adopters have a strong
voice in the decisions made by the OpenSFS Board. Your Community Board
Member is guided by Supporter and Adopter values as the Board makes
decisions for OpenSFS including how OpenSFS Development and Support funding
We'd like to thank Tommy Minyard from Texas Advanced Computing Center for
his excellent service as the 2013 Community Representative Director.
Full details regarding the nomination process can be found online:
Galen M. Shipman - Chairman, OpenSFS
Happy New Year everyone!
For our first 2014 post, we wanted to take a quick look back on the
accomplishments of last year. In 2013, OpenSFS added two new Participants --
Fujitsu and NCSA -- and Intel moved up to Promoter level and joined the
OpenSFS board. OpenSFS also pushed enhancements to our communications and
marketing efforts, added more mature management infrastructure, continued
funding the Lustre tree contract and added more Lustre development funding,
supported enhanced feature availability, and had a significant hand in
building a stronger foundation that the Lustre vendor ecosystem relies on
for their product lines. In particular, we continued to contribute
financially to Lustre. We spent over $3 million on Lustre technical
development and maintenance in 2013, our largest annual commitment to date.
Want to hear more? www.opensfs.org/lustre-transparency-momentum-2014/
Looking forward to working with you in 2014!
The whole OpenSFS organization
Hello OpenSFS participants,
Do you have opinions and suggestions for this year's LUG event? If so, we'd
love your input!
OpenSFS is forming the LUG 2014 planning committee and we're calling for
participants! The LUG planning committee is a vital group of volunteers
responsible for providing input, vetting presentation submissions, and
advising on event details. If you've attended LUG and have feedback on what
went well, suggested improvements, and speaking recommendations, we want to
hear from you!
On a company level, participation in the LUG planning committee will provide
insight to LUG content and sessions, allow your company to influence the
direction of LUG, and increase involvement in the Lustre community.
The LUG planning committee will meet 2-4 times a month in order to review
event plans, select presentation content, and ensure this year's LUG is a
smashing success. We look forward to participation from the broader Lustre
If you're interested in participating in the LUG planning committee, please
contact OpenSFS Administration with the following information by Monday,
* Time zone/physical location
As a reminder, this year's LUG will be held in Miami, Florida, April 8-10th.
Sponsorship opportunities <http://www.opensfs.org/lug-2014-sponsorship/>
and the call for presentations
<http://www.opensfs.org/lug-2014-call-papers/> are now open! Stay tuned for
Open Scalable File Systems, Inc. is a strong and growing nonprofit
organization dedicated to the success of the LustreR file system. OpenSFS
was founded in 2010 to advance Lustre development, ensuring it remains
vendor-neutral, open, and free <http://lustre.opensfs.org/download-lustre/>
. Since its inception, OpenSFS has been responsible for advancing the Lustre
file system and delivering new releases
<http://lustre.opensfs.org/community-lustre-roadmap/> on behalf of the open
source community. Through working groups, events, and ongoing funding
initiatives, OpenSFS harnesses the power of collaborative development to
fuel innovation and growth of the Lustre file system worldwide.
llog interface was redesigned and rebased on top of OSD between Lustre
2.1 and 2.4.
This is impacting Changelogs.
Now, in 2.4, changelog_catalog and changelog_users file are in root MDT
directory. In 2.1, there were in CONFIGS/ MDT directory.
Also, the changelog_catalogs were pointing to other files in OBJECTS/
directory. This directory does not exist anymore in Lustre 2.4
I'm not sure were those files are now, maybe in "0". There, filename
also have changed.
Am I correctly understanding the changes made in this area?
Was the file relocation something done in purpose?
Anyway, the upgrade case is not handle as a freshly updated MDT will
"forget" all the previous changelog users and changelog records. If this
is not know, I will open a ticket for that.