October 2014 - HPDD-discuss

Re: [HPDD-discuss] [Lustre-discuss] strange symbolic link behavior on mdt?

by Dilger, Andreas

On 2014/10/13, 12:22 PM, "Kristen J. Webb" <kwebb(a)teradactyl.com> wrote: >I am testing symbolic link behaviour >and not getting the expected results. > >In the simplest case, I create one valid >and one invalid link: > >-rw-r--r-- 1 root root 0 Oct 13 11:17 exist >lrwxrwxrwx 1 root root 5 Oct 13 11:17 lexist -> exist >lrwxrwxrwx 1 root root 7 Oct 13 11:17 lnoexist -> noexist > >When I look at the meta data through the ldiskfs mounted mdt >the link paths are not shown: > >-rw-r--r-- 1 root root 0 Oct 13 11:17 exist >lrwxrwxrwx 1 root root 5 Oct 13 11:17 lexist >lrwxrwxrwx 1 root root 7 Oct 13 11:17 lnoexist You shouldn't be mounting the MDT directly as ldiskfs while it is already mounted as Lustre. There is no attempt made to maintain cache coherency between the Lustre and ldiskfs mount points. Presumably the symbolic links worked fine on another Luster client? Cheers, Andreas >And trying to get a bit more info: > ># file * >exist: empty >lexist: unreadable symlink `lexist' (Invalid argument) >lnoexist: unreadable symlink `lnoexist' (Invalid argument) > > >If I restart lustre by unmounting and re-mounting >all the lustre mount points (MDT, ODT's, client) >the link paths show up in the ldiskfs mounted mdt: > >-rw-r--r-- 1 root root 0 Oct 13 11:17 exist >lrwxrwxrwx 1 root root 5 Oct 13 11:17 lexist -> exist >lrwxrwxrwx 1 root root 7 Oct 13 11:17 lnoexist -> noexist > >file * >exist: empty >lexist: symbolic link to `exist' >lnoexist: broken symbolic link to `noexist' > > >I've also just tried just re-mounting the ldiskfs mount >point but that did not help. > >Is there any way to correct this without having to restart lustre? > >I experimented with running lfsck w/o restart all of lustre >but I could not see that it was running or had done anything >useful yet. > >Kris >-- >This message is NOT encrypted >-------------------------------- >Mr. Kristen J. Webb >Chief Technology Officer >Teradactyl LLC. >2450 Baylor Dr. S.E. >Albuquerque, New Mexico 87106 >Phone: 1-505-338-6000 >Email: kwebb(a)teradactyl.com >Web: http://www.teradactyl.com > >Providers of Scalable Backup Solutions > for Unique Data Environments > >-------------------------------- >NOTICE TO RECIPIENTS: Any information contained in or attached to this >message >is intended solely for the use of the intended recipient(s). If you are >not the >intended recipient of this transmittal, you are hereby notified that you >received this transmittal in error, and we request that you please delete >and >destroy all copies and attachments in your possession, notify the sender >that >you have received this communication in error, and note that any review >or >dissemination of, or the taking of any action in reliance on, this >communication >is expressly prohibited. > > >Regular internet e-mail transmission cannot be guaranteed to be secure or >error-free. Therefore, we do not represent that this information is >complete or >accurate, and it should not be relied upon as such. If you prefer to >communicate >with Teradactyl LLC. using secure (i.e., encrypted and/or digitally >signed) >e-mail transmission, please notify the sender. Otherwise, you will be >deemed to >have consented to communicate with Teradactyl via regular internet e-mail >transmission. Please note that Teradactyl reserves the right to >intercept, >monitor, and retain all e-mail messages (including secure e-mail >messages) sent >to or from its systems as permitted by applicable law. > > > >_______________________________________________ >Lustre-discuss mailing list >Lustre-discuss(a)lists.lustre.org >http://lists.lustre.org/mailman/listinfo/lustre-discuss > Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division

6 years, 8 months

1
0
0 / 0

MDS OOMs with 2.5.3 clients and lru_size != 0

by David Singleton

We have seen mds (admittedly with smallish memory) OOM'ing while testing 2.5.3 whereas there was no problem with 2.5.0. It turns out the problem is that, even though we have lru_size=800 everywhere, the client LDLM lru's are growing huge so that the MDS unreclaimable ldlm slabs fill memory. It looks like the root cause is the change to ldlm_cancel_aged_policy() in commit 0a6c6fcd46 on the 2.5 branch (LU-4786 osc: to not pick busy pages for ELC) - it has changed the lru_sze != 0 behaviour. Prior to that, the non-lru_resize behaviour (at least through the early_lock_cancel path which is what we see being hit) was effectively cancel lock if (too many in lru cache || lock unused too long) In 2.5.3, it's cancel lock if (too many in lru cache && lock unused too long) Disabling early_lock_cancel doesn't seem to help. It might be arguable which of the two behaviours is correct but the lru_size doco suggests the former - the latter makes lru_size != 0 ineffective in practice. It also looks like the change was not actually necessary for LU-4300? Cheers David

6 years, 8 months

2
1
0 / 0

Running "rm -rf" causes high MDS memory usage

by Mohr Jr, Richard Frank (Rick Mohr)

I have run into a situation where a user's "rm -rf" process seems to cause very high buffer usage on our mds server. I verified that this process was the cause of the issue by sending the STOP signal to the "rm" command and noticing that the growth of the buffers on the mds server slowed to a crawl. If I then sent the CONT signal, the buffer size would start growing again. I dropped all the caches on the mds server and timed the growth. It increased about 17 GB in 15 mins. I have been dropping the caches periodically in an effort to contain the growth while I investigate the problem. Unfortunately, after I drop the caches, the new low point is always a little higher than it was before, which means there will come a point where dropping the caches will no longer be effective. With that in mind, I have had to stop the user's "rm" process to contain the damage. Since that is only a temporary band-aid, I am trying to get a handle on what might be the underlying problem. I searched the lustre bugs, and the closest thing I could find was LU-4740 (and maybe LU-4906?), but it's not clear if those are the cause of my problem. The odd thing is that a recursive rm is not an uncommon command to run, and I have not noticed this behavior before. The server is running: - CentOS 6.5 - kernel 2.6.32-358.23.2 - lustre 2.4.3 The client is running: - CentOS 6.2 - kernel 2.6.32-358.23.2 - lustre 1.8.9-wc1 Has anyone else seen a similar issue? -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu

6 years, 8 months

10
21
0 / 0

Lustre 2.7 update - October 10th 2014

by Jones, Peter A

http://wiki.opensfs.org/Lustre_2.7_-_October_10th_2014

6 years, 8 months

1
0
0 / 0

Question to build lustre from source

by Jorge Ventura

I would like to build lustre from source in way that be independent of distribution. I have a box with linux built from scratch and there is no specific distribution associated. What is the source kernel version that I have to use? Any special recommendation for parameters to be used for ./configure? Ventura

6 years, 8 months

4
8
0 / 0

It's Not Too Late to Register for CLUG

by OpenSFS Administration

<http://info.intel.com/rs/intel/images/HPDD%20Lustre%20User%20Group%20Event% 202.jpg> Hello- We are less than a week away from the LUG Event in Beijing on October 14th. It's not too late to register to join us at this conference. REGISTER NOW <http://pages.intel.com/kP0LZ0v00DA03Wl0l00w60Y> Our Sponsor List has expanded to include a few new names, and the speaker list is equally impressive. Platinum Sponsors * Inspur * Huawei * DataDirect Networks Gold Sponsors * Dell * Paratera * BingoCloud * Macrosan * STD Don't miss out, register today. We look forward to having you as our guest at the China Lustre User Group event. Regards, IntelR High Performance Data Team

6 years, 8 months

1
0
0 / 0

time until qos_prio_free=0 has an impact

by Michael Kluge

Hi all, we recently reactivated an OST and now the 450 nodes send this OST many more I/O requests that they send to the other osts. We have 4 servers and 48 osts. The other oss have a load of about 100. The server with this ost has 1500 and many "filter_commitrw_write()) scratch-OST002f: slow i_mutex 30s" messages. Thus, I set qos_prio_free to 0 in /proc/fs/lustre/lov/scratch-MDT0000-mdtlov Does anyone have an expectation how long it will take until the load on this server will go down? Hours? Days? Regards, Michael -- Dr.-Ing. Michael Kluge Technische Universität Dresden Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden Germany Contact: Willersbau, Room A 208 Phone: (+49) 351 463-34217 Fax: (+49) 351 463-37773 e-mail: michael.kluge(a)tu-dresden.de WWW: http://www.tu-dresden.de/zih

6 years, 8 months

4
7
0 / 0

Re: [HPDD-discuss] [Lustre-discuss] Lustre and OFED

by Simmons, James A.

>I see all the questions and ask myself: >Is the difference between OFED and the infiniband drivers in the CentOS repos tangible? It depends. I found for most cases it is not worth the extra effort to use the OFED/Mellanox stack. Now if you have FDR Mellanox cards then yes using the Mellanox/OFED 3.12 stack makes a difference. We did have one cluster that was having problems with the RHEL infiniband stack and going to the Mellanox stack helped in that case. >We had the OFED problems and just switched to the CentOS drivers, since all the clients are running with the Debian/kernel.org<http://kernel.org> drivers it didn't seem to make a big difference anyhow... > >>On Fri, Oct 3, 2014 at 9:40 AM, Indivar Nair <indivar.nair(a)techterra.in<mailto:indivar.nair@techterra.in>> wrote: >>Hi ..., >>I can see quite a few mails on Lustre and OFED. >> >>Are there any known issues with CentOS6.5+Lustre 2.5.3+Mellanox OFED 2.3 combination?

6 years, 8 months

3
2
0 / 0

Re: [HPDD-discuss] [Lustre-discuss] Lustre and OFED

by E.S. Rosenberg

I see all the questions and ask myself: Is the difference between OFED and the infiniband drivers in the CentOS repos tangible? We had the OFED problems and just switched to the CentOS drivers, since all the clients are running with the Debian/kernel.org drivers it didn't seem to make a big difference anyhow... On Fri, Oct 3, 2014 at 9:40 AM, Indivar Nair <indivar.nair(a)techterra.in> wrote: > Hi ..., > > I can see quite a few mails on Lustre and OFED. > > Are there any known issues with CentOS6.5+Lustre 2.5.3+Mellanox OFED 2.3 > combination? > > Regards, > > > Indivar Nair > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss(a)lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >

6 years, 8 months

1
0
0 / 0

lustre lnet infiniband config

by aayush agrawal

Hi, I am trying to build lustre 2.5.0 against MLNX_OFED_LINUX-2.2-1.0.1-rhel6.4-x86_64 on CentOS6.4 with kernel version 2.6.32-358. But I am not able to set lnet config settings properly. I used settings suggested in lustre 2.x manual. But then not able to get network up using lctl. Details: I have two server machines, one for mgs+mdt and second for oss and one client machine. I want to setup Infiniband on all these machines. I could run below steps successfully for all the three machines: 1. Run script mlnxofedinstall # ./mlnxofedinstall -vvv --add-kernel-support --without-32bit --without-fw-update --hpc 2. Restart openibd service # /etc/init.d/openibd restart 3. configure ib0 interface. 4. configure lustre with o2ib # ./configure --with-linux=Path_to_linux-2.6.32-358.18.1.el6 --with-o2ib=/usr/src/ofa_kernel/default/ 5. make lustre rpms: # make rpms This gave me below compilation error I looked online for this error and found bug registered on the same: https://jira.hpdd.intel.com/browse/LU-4266 Below patch from above link solved the problem and hence I could build lustre rpms: http://review.whamcloud.com/#/c/8451/1 Now first I want to do the Infiniband setup for mgs and mdt on single machine which also has Ethernet IP. Then I want to format and mount mgs and mdt. So I installed above created lustre rpms and then added below line in /etc/modprobe.d/lustre.conf options lnet networks=o2ib(ib0) Then I rebooted the machine to remove all lustre related modules including lnet and then ranmodprobe lnet command to add above parameters and the ran lctl network up which is giving me below error: LNET configure error 100: Network is down I looked online and found below discussion on same error: http://lists.lustre.org/pipermail/lustre-discuss/2010-June/013510.html As per suggestion in above mail I tried with below line in /etc/modprobe.d/lustre.conf. In below command for IB_IP, I have given infiniband IP. options lnet *networks=o2ib(ib0)* routes="tcp0 IB_IP@o2ib" This command hangs for around 2 to 3 minutes and then gives error: Write failed: Broken pipe. Same is the case for "options lnet *networks=o2ib(ib0)*" But if I set: options lnet *networks=tcp0(eth0),o2ib(ib0)* routes="tcp1 IB_IP@o2ib" then it gives LNET configure error 100: Network is down. It seems that for network=o2ib(ibo) I am getting error Write failed: Broken pipe. Am I missing anything while following above steps? Or how do I resolve above error? Thanks, Aayush.

6 years, 8 months

3
5
0 / 0

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

HPDD-discuss October 2014