lustre 2.5.0 + zfs
by luka leskovec
Hello all,
i got a running lustre 2.5.0 + zfs setup on top of centos 6.4 (the kernels
available on the public whamcloud site), my clients are on centos 6.5
(minor version difference, i recompiled the client sources with the options
specified on the whamcloud site)
but now i have some problems. I cannot judge how serious it is, as the only
problems i observe are slow responses on ls, rm and tar and apart from that
it works great. i also export it over nfs, which sometimes hangs the client
on which it is exported, but i expect this is an issue related to how many
service threads i have running on my servers (old machines).
but my osses (i got two) keep spitting out these messages into the system
log:
xxxxxxxxxxxxxxxxxxxxxx kernel: SPL: Showing stack for process 3264
xxxxxxxxxxxxxxxxxxxxxx kernel: Pid: 3264, comm: txg_sync Tainted:
P --------------- 2.6.32-358.18.1.el6_lustre.x86_64 #1
xxxxxxxxxxxxxxxxxxxxxx kernel: Call Trace:
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa01595a7>] ?
spl_debug_dumpstack+0x27/0x40 [spl]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa0161337>] ?
kmem_alloc_debug+0x437/0x4c0 [spl]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa0163b13>] ?
task_alloc+0x1d3/0x380 [spl]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa0160f8f>] ?
kmem_alloc_debug+0x8f/0x4c0 [spl]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa02926f0>] ? spa_deadman+0x0/0x120
[zfs]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa016432b>] ?
taskq_dispatch_delay+0x19b/0x2a0 [spl]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa0164612>] ?
taskq_cancel_id+0x102/0x1e0 [spl]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa028259a>] ? spa_sync+0x1fa/0xa80
[zfs]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa0295707>] ?
txg_sync_thread+0x307/0x590 [zfs]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffff810560a9>] ?
set_user_nice+0xc9/0x130
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa0295400>] ?
txg_sync_thread+0x0/0x590 [zfs]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa0162478>] ?
thread_generic_wrapper+0x68/0x80 [spl]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffffa0162410>] ?
thread_generic_wrapper+0x0/0x80 [spl]
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
xxxxxxxxxxxxxxxxxxxxxx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
does anyone know, is this a serious problem, or just aesthetics? any way to
solve this? any hints?
best regards,
Luka Leskovec
6 years, 10 months
Re: [HPDD-discuss] iozone and lustre
by E.S. Rosenberg
On Thu, Feb 20, 2014 at 7:46 PM, Alexander I Kulyavtsev <aik(a)fnal.gov>wrote:
> You may want to use iozone in throughput mode, with flag -t. It will use
> multple writers/readers on the same node.
>
> If you wish to use several client nodes, use iozone cluster mode -+m . You
> need to prepare cluster file to describe file locations.
>
> Use file size , data set large enough to "flush" data after write on
> server, otherwise read rates are "too good" as data read from memory. Try
> iozone remount option. We used iozone wrapper on slave nodes and set "dont
> need" flag for the file on write so it is flushed from cache, at least
> rates changed.
>
> You may want to watch rates on network and choose time period when all
> processes do io. Iozone printout includes period when some procs finished
> and late starters doing io.
> But: network rates include retries and read aheads.
>
> This is independent of file striping.
>
Thanks Alex I'm going to try that!
Sorry that I keep bothering everyone but:
How do I troubleshoot lfs setstripe that refuses saying it's getting an
invalid argument (22)?
lfs setstripe iozone.tmp -c 10
unable to open 'iozone.tmp': Invalid argument (22)
error: setstripe: create stripe file 'iozone.tmp' failed
I have tried first touching the file, I have tried it on directories (both
existing and non-existing) but all the same result.
I am running the command as a regular user inside a folder that belongs to
the user....
Thanks,
Eli
>
> Alex
>
> On Feb 20, 2014, at 8:01 AM, "E.S. Rosenberg" <
> esr+hpdd-discuss(a)mail.hebrew.edu> wrote:
>
> > at the moment I am just using automated tests...
> >
> > (iozone -Ra -g 64G)
>
6 years, 12 months
filesystem usable space
by Javed Shaikh
hi,
specs: lustre 2.4.2 (client and server)
where can i find more info on /proc/fs/lustre/llite/blah-file-system/kbytes* files? their purpose.
basically i’m interested in getting actual usable space on filesystem … using df (or lfs df) if i add up ‘avail’ + ‘used’ space, it doesn’t give ‘size’ … so its known the filesystem is using some space for itself, but where is this info stored?
any pointers will be helpful.
thanks,
javed
7 years
Top 5 Reasons to Attend LUG 2014
by OpenSFS Administration
<http://www.opensfs.org/events/lug14/>
Lustre User Group 2014
Miami Marriott Biscayne Bay, April 8-10
On April 8-10th, OpenSFS is hosting <http://www.opensfs.org/lug14/> the
12th Annual LustreR User Group (LUG) Conference at the
<http://www.marriott.com/hotels/travel/miabb-miami-marriott-biscayne-bay/>
Miami Marriott Biscayne Bay in Miami, Florida! LUG 2014 will bring together
industry leaders, end-users, developers, and vendors to talk Lustre,
contribute to the community, and move the technology forward.
Here are the top 5 reasons why you should attend LUG 2014:
5 - Technical Poster Exhibition so you can digest Lustre info on your own
schedule
Includes posters on Lustre challenges, configurations, data movement between
Lustre and enterprise storage, file system design, and many more.
4 - Located minutes from South Beach so you can work on your tan
The venue for LUG 2014 is the Miami Marriott Biscayne Bay which is located
right on Biscayne Bay and just minutes from South Beach and other famous
Miami attractions.
3 - Best place to hear about feature development so you can be ahead of the
curve
Talks on Lustre roadmap, Lustre performance comparison and tuning, Exascale,
proposed new features, Hadoop, log analysis, and much more. The full 2014
LUG agenda is now available <http://www.opensfs.org/lug-2014-agenda/>
online.
2 - First Lustre Annual Report so you can see how Lustre is doing in new
markets
OpenSFS is sponsoring its first Lustre Annual Report that will examine the
current status of Lustre in multiple vertical markets spanning high
performance computing, Big Data, and enterprise computing markets, including
the drivers and barriers for future adoption.
1 - Best event of the year to talk Lustre so you can be a true insider
Finally a place where everyone understands Lustre! In fact, they might know
more than you. LUG is the best event of the year, not just for the
presentations, but also for the hallway discussions and the evening
get-togethers, too.
<https://opensfs.wufoo.com/forms/lug-conference-2014/> Register now to take
advantage of the special early bird rate before March 17th. The early bird
registration price is $525.
Be sure to check back regularly, as speakers and session highlights will
also be posted over the coming weeks. In the meantime, the LUG 2014 agenda
is available <http://www.opensfs.org/lug-2014-agenda/> online.
<https://opensfs.wufoo.com/forms/lug-conference-2014/> Register today for
LUG 2014! If you have any questions, please feel free to contact
<mailto:admin@opensfs.org> admin(a)opensfs.org.
Best regards,
OpenSFS LUG Planning Committee
_________________________
OpenSFS Administration
3855 SW 153rd Drive Beaverton, OR 97006 USA
Phone: +1 503-619-0561 | Fax: +1 503-644-6708
Twitter: <https://twitter.com/opensfs> @OpenSFS
Email: <mailto:admin@opensfs.org> admin(a)opensfs.org | Website:
<http://www.opensfs.org> www.opensfs.org
<http://www.opensfs.org/lug-2014-sponsorship/>
<http://www.opensfs.org/lug-2014-sponsorship/> Click here to learn how to
become a 2014 LUG Sponsor
Open Scalable File Systems, Inc. was founded in 2010 to advance Lustre
development, ensuring it remains vendor-neutral, open, and free. Since its
inception, OpenSFS has been responsible for advancing the Lustre file system
and delivering new releases on behalf of the open source community. Through
working groups, events, and ongoing funding initiatives, OpenSFS harnesses
the power of collaborative development to fuel innovation and growth of the
Lustre file system worldwide.
7 years
Lustre Survey 2014 by OpenSFS
by Christopher J. Morrone
Dear Lustre Community,
The OpenSFS Community Development Working Group is gathering data from
oranizations using Lustre in order to develop a long-term support
strategy recommendation for Lustre.
We want to ensure that future Lustre releases are well-aligned with the
needs of the Lustre community.
Please complete this short survey to make sure that your organization's
voice is heard!
https://www.surveymonkey.com/s/P3HFY7R
Note that all questions are optional so it is ok to submit a partially
completed survey if you prefer not to disclose some information.
Best regards,
Christopher Morrone
OpenSFS CDWG Lead
7 years
FW: Time to kill off some old LND driver
by Dilger, Andreas
Forwarding this to lustre-discuss and the mxlnd author to reach the
maximum audience.
I definitely think qswlnd is beyond its useful lifetime (Quadrics is out
of business since 2009, so any systems using it are at least 5 years old),
and I believe ralnd is the same boat (it was only used for much older Cray
systems). No idea if anyone is still using mxlnd?
On 2014/02/20, 9:08 AM, "Simmons, James A." <simmonsja(a)ornl.gov> wrote:
>Currently the main LND drivers in the tree are o2iblnd, gnilnd, and
>socklnd. We also have a few drivers which are up for discussion for
>removal.
>The drivers up for discussion for removal are:
>
>qswlnd
>mxlnd
>ralnd
>
>If anyone is using these drivers speak up otherwise their removal will be
>put into the pipeline.
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
7 years
Time to kill off some old LND driver
by Simmons, James A.
Currently the main LND drivers in the tree are o2iblnd, gnilnd, and socklnd. We also have a few drivers which are up for discussion for removal.
The drivers up for discussion for removal are:
qswlnd
mxlnd
ralnd
If anyone is using these drivers speak up otherwise their removal will be put into the pipeline.
7 years
iozone and lustre
by E.S. Rosenberg
Hi all,
When I run the iozone benchmark on lustre it ends up only using one OSS
server, I assume this has something to do with the way it writes but I'm
not really sure where to look, at the moment I am just using automated
tests...
(iozone -Ra -g 64G)
lustre is functioning fine and when using dd from multiple machines with a
bs=102400 it uses all OSS.
Any pointers would be very welcome...
Thanks,
Eli
7 years
filefrag and file logical order
by John Bauer
How do I determine the file logical order of the Lustre extents, given
the device logical order? Can I assume the file logical order of the
extents will follow the
the device order indicated by obdidx from the "lfs getstripe" output?
Is it safe to assume that the order of the devices as discovered in
the filefrag output is always the same as the order of the devices in
the "lfs getstripe" output? In the output below, both have the order 4,
12, 9, 1. Is this coincidental or is it a given?
% lfs getstripe -v /lus/scratch/p01940/pwrite.dat
/lus/scratch/p01940/pwrite.dat
lmm_magic: 0x0BD10BD0
lmm_seq: 0x2002dab80
lmm_object_id: 0xd1e
lmm_stripe_count: 4
lmm_stripe_size: 4194304
lmm_stripe_pattern: 1
lmm_layout_gen: 0
lmm_stripe_offset: 4
obdidx objid objid group
4 845042 0xce4f2 0
12 845220 0xce5a4 0
9 844466 0xce2b2 0
1 844523 0xce2eb 0
% /usr/sbin/filefrag -v /lus/scratch/p01940/pwrite.dat
Filesystem type is: bd00bd0
File size of /lus/scratch/p01940/pwrite.dat is 4194304000 (4096000
blocks of 1024 bytes)
ext: device_logical: physical_offset: length: dev: flags:
0: 0.. 10239: 1850072064..1850082303: 10240: 0004: network
.
115: 928768.. 1023999: 1851001856..1851097087: 95232: 0004: network
116: 0.. 16383: 14315287552..14315303935: 16384: 000c: network
.
270: 1021952.. 1023999: 14316310528..14316312575: 2048: 000c: network
271: 0.. 10239: 4657097728..4657107967: 10240: 0009: network
.
381: 945152.. 946175: 4658040832..4658041855: 1024: 0009: network
431: 978944.. 1023999: 4333852672..4333897727: 45056: 0001: network
/lus/scratch/p01940/pwrite.dat: 421 extents found
%
7 years
