1:25 p.m.
Hi,
I have been playing around with setting vfs_cache_pressure=0 on our v2.4.1 MDS and noticed
that the dentry usage as reported from slabtop looks very odd. It seems to remain pretty
much the same for long periods of time and then jumps up suddenly. I have attached a graph
of ldiskfs_inode_cache,dentry and buffer_head.
https://db.tt/DSxKl5gm
Is the dentry usage tracked in some other slab? The server ran out of memory not long
after the dentry count jumped up. What I'm interesting in doing is track slab usage
and toggle vfs_cache_pressure to avoid running out of memory whilst maximising the
inode/dentry cache.
Daire
1:28 p.m.
are you running with MDS + client on the same node?
thanks, Alex
On Dec 4, 2013, at 4:25 PM, Daire Byrne <daire(a)dneg.com> wrote:
Hi,
I have been playing around with setting vfs_cache_pressure=0 on our v2.4.1 MDS and
noticed that the dentry usage as reported from slabtop looks very odd. It seems to remain
pretty much the same for long periods of time and then jumps up suddenly. I have attached
a graph of ldiskfs_inode_cache,dentry and buffer_head.
https://db.tt/DSxKl5gm
Is the dentry usage tracked in some other slab? The server ran out of memory not long
after the dentry count jumped up. What I'm interesting in doing is track slab usage
and toggle vfs_cache_pressure to avoid running out of memory whilst maximising the
inode/dentry cache.
Daire<mds-slab.png>_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
2:01 p.m.
Ah yes... I was doing that for testing something else and forgot to unmount. Does that
explain the large jump?
----- Original Message -----
are you running with MDS + client on the same node?
thanks, Alex
On Dec 4, 2013, at 4:25 PM, Daire Byrne <daire(a)dneg.com> wrote:
> Hi,
>
> I have been playing around with setting vfs_cache_pressure=0 on our v2.4.1
> MDS and noticed that the dentry usage as reported from slabtop looks very
> odd. It seems to remain pretty much the same for long periods of time and
> then jumps up suddenly. I have attached a graph of
> ldiskfs_inode_cache,dentry and buffer_head.
>
> https://db.tt/DSxKl5gm
>
> Is the dentry usage tracked in some other slab? The server ran out of
> memory not long after the dentry count jumped up. What I'm interesting in
> doing is track slab usage and toggle vfs_cache_pressure to avoid running
> out of memory whilst maximising the inode/dentry cache.
>
> Daire<mds-slab.png>_______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
2:34 p.m.
well, generally MDS (since 2.0) shouldn’t be using dentry structure much.
thanks, Alex
On Dec 4, 2013, at 5:01 PM, Daire Byrne <daire(a)dneg.com> wrote:
Ah yes... I was doing that for testing something else and forgot to
unmount. Does that explain the large jump?
----- Original Message -----
> are you running with MDS + client on the same node?
>
> thanks, Alex
>
> On Dec 4, 2013, at 4:25 PM, Daire Byrne <daire(a)dneg.com> wrote:
>
>> Hi,
>>
>> I have been playing around with setting vfs_cache_pressure=0 on our v2.4.1
>> MDS and noticed that the dentry usage as reported from slabtop looks very
>> odd. It seems to remain pretty much the same for long periods of time and
>> then jumps up suddenly. I have attached a graph of
>> ldiskfs_inode_cache,dentry and buffer_head.
>>
>> https://db.tt/DSxKl5gm
>>
>> Is the dentry usage tracked in some other slab? The server ran out of
>> memory not long after the dentry count jumped up. What I'm interesting in
>> doing is track slab usage and toggle vfs_cache_pressure to avoid running
>> out of memory whilst maximising the inode/dentry cache.
>>
>> Daire<mds-slab.png>_______________________________________________
>> HPDD-discuss mailing list
>> HPDD-discuss(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>
>
> --------------------------------------------------------------------
> Closed Joint Stock Company Intel A/O
> Registered legal address: Krylatsky Hills Business Park,
> 17 Krylatskaya Str., Bldg 4, Moscow 121614,
> Russian Federation
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
2:49 p.m.
Alexey,
So is there another structure that dirs are cached in that is effected by
vfs_cache_pressure=0? I'm trying to understand why the server ran out of memory even
though ldiskfs_inode_cache was only 131G at the time (in a 400G server). I'm not
currently monitoring all slabinfo entries so I can't say which others had grown very
large.
Cheers,
Daire
----- Original Message -----
well, generally MDS (since 2.0) shouldn’t be using dentry structure
much.
thanks, Alex
On Dec 4, 2013, at 5:01 PM, Daire Byrne <daire(a)dneg.com> wrote:
> Ah yes... I was doing that for testing something else and forgot to
> unmount. Does that explain the large jump?
>
> ----- Original Message -----
>> are you running with MDS + client on the same node?
>>
>> thanks, Alex
>>
>> On Dec 4, 2013, at 4:25 PM, Daire Byrne <daire(a)dneg.com> wrote:
>>
>>> Hi,
>>>
>>> I have been playing around with setting vfs_cache_pressure=0 on our
>>> v2.4.1
>>> MDS and noticed that the dentry usage as reported from slabtop looks very
>>> odd. It seems to remain pretty much the same for long periods of time and
>>> then jumps up suddenly. I have attached a graph of
>>> ldiskfs_inode_cache,dentry and buffer_head.
>>>
>>> https://db.tt/DSxKl5gm
>>>
>>> Is the dentry usage tracked in some other slab? The server ran out of
>>> memory not long after the dentry count jumped up. What I'm interesting
in
>>> doing is track slab usage and toggle vfs_cache_pressure to avoid running
>>> out of memory whilst maximising the inode/dentry cache.
>>>
>>> Daire<mds-slab.png>_______________________________________________
>>> HPDD-discuss mailing list
>>> HPDD-discuss(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>>
>>
>> --------------------------------------------------------------------
>> Closed Joint Stock Company Intel A/O
>> Registered legal address: Krylatsky Hills Business Park,
>> 17 Krylatskaya Str., Bldg 4, Moscow 121614,
>> Russian Federation
>>
>> This e-mail and any attachments may contain confidential material for
>> the sole use of the intended recipient(s). Any review or distribution
>> by others is strictly prohibited. If you are not the intended
>> recipient, please contact the sender and delete all copies.
>>
>>
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
2:53 p.m.
it’s not that easy now.. we do not cache individual directory entries (like dcache does)
anymore.
which is probably subject to discussion and further optimization. but buffer cache is
still used to
hold directory blocks.
probably it makes sense to watch pure MDS first (i.e. do not use Lustre client on MDS
node).
thanks, Alex
On Dec 4, 2013, at 5:49 PM, Daire Byrne <daire(a)dneg.com> wrote:
Alexey,
So is there another structure that dirs are cached in that is effected by
vfs_cache_pressure=0? I'm trying to understand why the server ran out of memory even
though ldiskfs_inode_cache was only 131G at the time (in a 400G server). I'm not
currently monitoring all slabinfo entries so I can't say which others had grown very
large.
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
2:29 p.m.
On Wed, Dec 04, 2013 at 12:25:06PM +0000, Daire Byrne wrote:
I have been playing around with setting vfs_cache_pressure=0 on our
v2.4.1 MDS ...
maybe 2.4 has changed something (I last comprehensively tested with
1.8), but that seems like an odd thing to try to me...
MDS's rarely have any VM activity going on that isn't Lustre inodes and
dentries and so (in my experience) perform well (and safely) with just
the default vfs_cache_pressure=100 setting.
IMHO it's on OSS's where inodes/dentries fare badly in competition with
Lustre's read and write-through caches where vfs_cache_pressure=0 (used
with caution) can be a huge win.
cheers,
robin
2:43 p.m.
Robin,
Well our Lustre filesystem has around 500 million files in it and at least half of those
are accessed every evening - a unique workload for sure (backups). Even with
vfs_cache_pressure=1 we don't seem to do a very good job of keeping a high enough
proportion of those in memory and our RAID array is taking quite a hammering seeking here
and there every day. We have 400Gigs in our servers so we'd like to put it to
effective use.
Also looking at the odd dentry graph I sent, I wonder why our inodes increase but the
dentries do not follow a similar (but scaled down) trend. In our normal (xfs+nfs) Linux
filers the dentry cache follows the inode cache trend very closely. To read 100 files you
traverse 4 dirs say.... So every file/inode will touch x number of dirs.
Daire
----- Original Message -----
On Wed, Dec 04, 2013 at 12:25:06PM +0000, Daire Byrne wrote:
>I have been playing around with setting vfs_cache_pressure=0 on our v2.4.1
>MDS ...
maybe 2.4 has changed something (I last comprehensively tested with
1.8), but that seems like an odd thing to try to me...
MDS's rarely have any VM activity going on that isn't Lustre inodes and
dentries and so (in my experience) perform well (and safely) with just
the default vfs_cache_pressure=100 setting.
IMHO it's on OSS's where inodes/dentries fare badly in competition with
Lustre's read and write-through caches where vfs_cache_pressure=0 (used
with caution) can be a huge win.
cheers,
robin
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
3089
days inactive
3089
days old
7 comments
3 participants
participants (3)
-
Alexey Zhuravlev -
Daire Byrne -
Robin Humble