12:34 p.m.
I'd like to announce the availability of Lester, the Lustre lister. It
is available from github at https://github.com/ORNL-TechInt/lester
Lester is an extension of (n)e2scan for generating lists of files (and
potentially their attributes) from a ext2/ext3/ext4/ldiskfs filesystem.
We primarily use it for generating a purge candidate list, but it is
also useful for generating a list of files affected by an OST outage or
providing a name for an inode.
For example, to list files that have not been accessed in two weeks and
put the output in ne2scan format in $OUTFILE:
touch -d 'now - 2 weeks' /tmp/flag
lester -A fslist -a before=/tmp/flag -o $OUTFILE $BLOCKDEV
To do the same thing, but generate a full listing of the filesystem in
parallel:
touch -d 'now - 2 weeks' /tmp/flag
lester -A fslist -a before=/tmp/flag -a genhit=$UNACCESSED_LIST
\
-o $FULL_LIST $BLOCKDEV
To name inodes to stdout (when not using Lustre 2.4's LINKEA):
lester -A namei -a $INODE1 -a $INODE2 ... $BLOCKDEV
To get a list of files with objects on OSTs 999 and 1000:
lester -A lsost -a 999 -a 1000 -o $OUTFILE $BLOCKDEV
To get a list of options and actions, use 'lester -h'; to get a list of
options for a given action, use 'lester -A $ACTION -a help'.
Lester uses its own AIO-based IO engine by default, which is usually
much faster than the default Unix engine for large filesystems on
high-performance devices. The number of requests in flight, request
size, cache size, and read-ahead settings for various phases of the scan
are all configurable. I recommend experimenting with the settings to
find a balance between speed and resource usage for your situation.
More information about the gains we've seen in testing prototype version
of Lester are in my LUG 2011 presentation,
http://www.opensfs.org/wp-content/uploads/2012/12/500-530_David_Dillow_LU...
--
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
10:15 a.m.
Hi David,
I have Lester compiled and tried running it against one of our Lustre MDTs.
It appears from the messages that our MDT was formatted with an incompatible feature? Is
this true, and if so, is there anything we can do about it?
Thanks,
Shawn
[root@hpclam00 lester-master]# ./lester -A lsost -a 0 /dev/mapper/vg_lustrea-mdt
ext2fs_open: Filesystem has unsupported feature(s) opening /dev/mapper/vg_lustrea-mdt
[root@hpclam00 lester-master]# dumpe2fs -h /dev/mapper/vg_lustrea-mdt
dumpe2fs 1.42.7.wc1 (12-Apr-2013)
Filesystem volume name: lustrea-MDT0000
Last mounted on: /
Filesystem UUID: 7db8ce81-9493-4deb-bc90-54e24627ef59
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype
needs_recovery mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink
quota
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 1572864000
Block count: 786432000
Reserved block count: 39321600
Free blocks: 589593672
Free inodes: 1572863653
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1024
Blocks per group: 16384
Fragments per group: 16384
Inodes per group: 32768
Inode blocks per group: 4096
RAID stripe width: 256
Flex block group size: 16
Filesystem created: Tue Dec 10 10:43:13 2013
Last mount time: Fri Jan 31 11:37:54 2014
Last write time: Fri Jan 31 11:37:54 2014
Mount count: 8
Maximum mount count: -1
Last checked: Tue Dec 10 10:43:13 2013
Check interval: 0 (<none>)
Lifetime writes: 755 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 512
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: c5595a63-e2ae-46a0-842e-17d4ba090c63
Journal backup: inode blocks
MMP block number: 13726
MMP update interval: 5
User quota inode: 3
Group quota inode: 4
Journal features: journal_incompat_revoke
Journal size: 400M
Journal length: 102400
Journal sequence: 0x0000c3b2
Journal start: 1
On Dec 30, 2013, at 2:34 PM, David Dillow <dave(a)thedillows.org> wrote:
I'd like to announce the availability of Lester, the Lustre
lister. It
is available from github at https://github.com/ORNL-TechInt/lester
Lester is an extension of (n)e2scan for generating lists of files (and
potentially their attributes) from a ext2/ext3/ext4/ldiskfs filesystem.
We primarily use it for generating a purge candidate list, but it is
also useful for generating a list of files affected by an OST outage or
providing a name for an inode.
For example, to list files that have not been accessed in two weeks and
put the output in ne2scan format in $OUTFILE:
touch -d 'now - 2 weeks' /tmp/flag
lester -A fslist -a before=/tmp/flag -o $OUTFILE $BLOCKDEV
To do the same thing, but generate a full listing of the filesystem in
parallel:
touch -d 'now - 2 weeks' /tmp/flag
lester -A fslist -a before=/tmp/flag -a genhit=$UNACCESSED_LIST
\
-o $FULL_LIST $BLOCKDEV
To name inodes to stdout (when not using Lustre 2.4's LINKEA):
lester -A namei -a $INODE1 -a $INODE2 ... $BLOCKDEV
To get a list of files with objects on OSTs 999 and 1000:
lester -A lsost -a 999 -a 1000 -o $OUTFILE $BLOCKDEV
To get a list of options and actions, use 'lester -h'; to get a list of
options for a given action, use 'lester -A $ACTION -a help'.
Lester uses its own AIO-based IO engine by default, which is usually
much faster than the default Unix engine for large filesystems on
high-performance devices. The number of requests in flight, request
size, cache size, and read-ahead settings for various phases of the scan
are all configurable. I recommend experimenting with the settings to
find a balance between speed and resource usage for your situation.
More information about the gains we've seen in testing prototype version
of Lester are in my LUG 2011 presentation,
http://www.opensfs.org/wp-content/uploads/2012/12/500-530_David_Dillow_LU...
--
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
8:10 p.m.
On Fri, 2014-01-31 at 18:15 +0000, Hall, Shawn wrote:
I have Lester compiled and tried running it against one of our
Lustre
MDTs.
It appears from the messages that our MDT was formatted with an
incompatible feature? Is this true, and if so, is there anything we
can do about it?
Hmm, I suppose it is possible, but since you are using a recent
libext2fs from Intel, I don't think it should be an issue.
Could you check 'ldd lester' vs 'ldd dumpe2fs' and see if they are
getting different libraries? My best guess at the moment is that lester
was compiled against the system libext2fs, which could present a
problem.
I'll try to check other avenues in parallel, but I don't currently have
any access to a Lustre file system, so I may be a bit slower than I
like.
10:42 p.m.
On Fri, 2014-01-31 at 18:15 +0000, Hall, Shawn wrote:
Filesystem features: ... mmp ...
This may be related; I've not tested with the MMP feature, but it really
shouldn't be an issue -- Lustre's libext2fs should be fine with that.
Actually, looking at debugfs's code, I see a test we should try.
In lester.c, function run_scan(), where it says
EXT2_FLAG_SOFTSUPP_FEATURES in the ext2fs_open() call, add
'| EXT2_FLAG_SKIP_MMP' so it gets both options.
I suspect this will fix the issue for you; let me know if it does and
I'll work up a proper patch.
10:22 a.m.
Thanks David,
I am trying to put lester on a shared file system so we can use it on our multiple file
systems, so I’ve been having more trouble than most because I’m trying to get all the
dependencies to work on a shared file system rather than installing locally.
I’ve been using the sourceforge vanilla e2fsprogs as DDN does not seem so provide a
e2fsprogs-devel package with their ISO. That may be part of the problem. Is
git.whamcloud.com the best place to retrieve the e2fsprogs code for this purpose? I was
having trouble getting snapshots off the website (downloading empty files), but I can try
using a git command line client to get it.
I tried the change to run_scan() like you suggested but that didn’t seem to help.
Shawn
On Feb 11, 2014, at 12:42 AM, David Dillow <dave(a)thedillows.org> wrote:
On Fri, 2014-01-31 at 18:15 +0000, Hall, Shawn wrote:
> Filesystem features: ... mmp ...
This may be related; I've not tested with the MMP feature, but it really
shouldn't be an issue -- Lustre's libext2fs should be fine with that.
Actually, looking at debugfs's code, I see a test we should try.
In lester.c, function run_scan(), where it says
EXT2_FLAG_SOFTSUPP_FEATURES in the ext2fs_open() call, add
'| EXT2_FLAG_SKIP_MMP' so it gets both options.
I suspect this will fix the issue for you; let me know if it does and
I'll work up a proper patch.
10:31 a.m.
On Feb 12, 2014, at 1:22 PM, "Hall, Shawn" <Shawn.Hall(a)bp.com>
wrote:
I’ve been using the sourceforge vanilla e2fsprogs as DDN does not
seem so provide a e2fsprogs-devel package with their ISO. That may be part of the
problem. Is git.whamcloud.com the best place to retrieve the e2fsprogs code for this
purpose?
Is this the download URL you were using?
http://downloads.whamcloud.com/public/e2fsprogs/
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
11:13 a.m.
Thanks Rick. I had never stumbled across that tree on downloads.whamcloud.com. I
installed the e2fsprogs-devel rpm and that did the trick. For reference, I did not have
to change the flags in lester.c either - the vanilla lester code works correctly if you
have the Intel e2fsprogs-devel package installed.
Thanks everyone!
Shawn
On Feb 12, 2014, at 12:31 PM, Mohr Jr, Richard Frank (Rick Mohr) <rmohr(a)utk.edu>
wrote:
On Feb 12, 2014, at 1:22 PM, "Hall, Shawn" <Shawn.Hall(a)bp.com>
wrote:
> I’ve been using the sourceforge vanilla e2fsprogs as DDN does not seem so provide a
e2fsprogs-devel package with their ISO. That may be part of the problem. Is
git.whamcloud.com the best place to retrieve the e2fsprogs code for this purpose?
Is this the download URL you were using?
http://downloads.whamcloud.com/public/e2fsprogs/
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
3:59 p.m.
So just some background - we run a full ls on each of our file systems every night in
order to know about the files our users have on the file systems, with file size being of
particular importance (so we know who the biggest space offenders are), so Lester looks
interesting to us.
From playing with it a little bit, it appears that Lester pulls what
it can from the MDT. It appears that file size is not one of these things, which makes
sense. It does appear that there may be something configurable by some like a “lctl
conf_param $FSNAME.mdt.som” on the MDT, which puts file sizes on metadata (SOM). Not much
documentation on this that I can find, but is this something that could be used by Lester
to find file sizes?
My other option is to try out Robinhood and see if that will do what we’re looking for as
well, which I think it will.
Thanks,
Shawn
On Feb 12, 2014, at 1:13 PM, Hall, Shawn <Shawn.Hall(a)bp.com> wrote:
Thanks Rick. I had never stumbled across that tree on
downloads.whamcloud.com. I installed the e2fsprogs-devel rpm and that did the trick. For
reference, I did not have to change the flags in lester.c either - the vanilla lester code
works correctly if you have the Intel e2fsprogs-devel package installed.
Thanks everyone!
Shawn
On Feb 12, 2014, at 12:31 PM, Mohr Jr, Richard Frank (Rick Mohr) <rmohr(a)utk.edu>
wrote:
>
> On Feb 12, 2014, at 1:22 PM, "Hall, Shawn" <Shawn.Hall(a)bp.com>
> wrote:
>
>> I’ve been using the sourceforge vanilla e2fsprogs as DDN does not seem so provide
a e2fsprogs-devel package with their ISO. That may be part of the problem. Is
git.whamcloud.com the best place to retrieve the e2fsprogs code for this purpose?
>
> Is this the download URL you were using?
>
> http://downloads.whamcloud.com/public/e2fsprogs/
>
> --
> Rick Mohr
> Senior HPC System Administrator
> National Institute for Computational Sciences
> http://www.nics.tennessee.edu
>
>
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:51 p.m.
Shawn,
If you're using a newer lustre with changelogs Robinhood has a very nice
feature in that it supports changelogs. This means you get a fairly
close to realtime picture of your filesystem, unlike routine scans.
For us, that's a great thing.
Scott
On 2/12/14 5:59 PM, Hall, Shawn wrote:
So just some background - we run a full ls on each of our file
systems every night in order to know about the files our users have
on the file systems, with file size being of particular importance
(so we know who the biggest space offenders are), so Lester looks
interesting to us.
From playing with it a little bit, it appears that Lester pulls what
it can from the MDT. It appears that file size is not one of these
things, which makes sense. It does appear that there may be
something configurable by some like a “lctl conf_param
$FSNAME.mdt.som” on the MDT, which puts file sizes on metadata (SOM).
Not much documentation on this that I can find, but is this something
that could be used by Lester to find file sizes?
My other option is to try out Robinhood and see if that will do what
we’re looking for as well, which I think it will.
Thanks, Shawn
On Feb 12, 2014, at 1:13 PM, Hall, Shawn<Shawn.Hall(a)bp.com> wrote:
> Thanks Rick. I had never stumbled across that tree on
> downloads.whamcloud.com. I installed the e2fsprogs-devel rpm and
> that did the trick. For reference, I did not have to change the
> flags in lester.c either - the vanilla lester code works correctly
> if you have the Intel e2fsprogs-devel package installed.
>
> Thanks everyone!
>
> Shawn
>
> On Feb 12, 2014, at 12:31 PM, Mohr Jr, Richard Frank (Rick
> Mohr)<rmohr(a)utk.edu> wrote:
>
>>
>> On Feb 12, 2014, at 1:22 PM, "Hall, Shawn"<Shawn.Hall(a)bp.com>
>> wrote:
>>
>>> I’ve been using the sourceforge vanilla e2fsprogs as DDN does
>>> not seem so provide a e2fsprogs-devel package with their ISO.
>>> That may be part of the problem. Is git.whamcloud.com the best
>>> place to retrieve the e2fsprogs code for this purpose?
>>
>> Is this the download URL you were using?
>>
>> http://downloads.whamcloud.com/public/e2fsprogs/
>>
>> -- Rick Mohr Senior HPC System Administrator National Institute
>> for Computational Sciences http://www.nics.tennessee.edu
>>
>>
>
> _______________________________________________ HPDD-discuss
> mailing list HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________ HPDD-discuss mailing
list HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
6:49 a.m.
So just some background - we run a full ls on each of our file systems
every night in order to know about the files our users have on the file systems, with file
>size being of particular importance (so we know who the biggest space offenders are),
so Lester looks interesting to us.
From playing with it a little bit, it appears that Lester pulls what it can from the MDT.
It appears that file size is not one of these things, which makes sense. It >does
appear that there may be something configurable by some like a "lctl conf_param
$FSNAME.mdt.som" on the MDT, which puts file sizes on metadata >(SOM). Not much
documentation on this that I can find, but is this something that could be used by Lester
to find file sizes?
SOM equals size on metadata which is not supported. Currently you have to scan the OSS to
figure out how much disk space is being used.
2850
days inactive
2895
days old
9 comments
5 participants
participants (5)
-
David Dillow -
Hall, Shawn -
Mohr Jr, Richard Frank (Rick Mohr) -
Scott Nolin -
Simmons, James A.