We've been having a devil of a time figuring out what's been going on with
what appears to be a memory leak in the Lustre page cache.
Some specifics: we are running Lustre 2.1.2, on RHEL 6.3, with kernel
2.6.32-279.11.1.el6. The symptoms are we end up with a system with all of
the memory consumed by pages on the inactive free list. Here's what
the relevant lines from /proc/meminfo look like:
MemTotal: 32914040 kB
MemFree: 9382100 kB
Active(file): 33348 kB
Inactive(file): 19620564 kB
What happens is a couple of users seem to be triggering this behavior; exactly
what causes it is still unclear, but one user triggers it just taring up
and removing some files. Eventually the inactive list keeps growing and
the OOM killer will come along, kill off various things in a futile
attempt to get rid of memory, and then the system will need to be rebooted.
I can't claim to be an expert on the Linux VM system, but I decided to
try my best at digging into this some more. I figured out where the
inactive page list was in memory and started looking at the pages using
"crash". There were over 5 million entries on the inactive page list,
so obviously I didn't look at them all; I picked a few at random and looked
at them. What I found was:
- They were all pages owned by Lustre (their backing store pointer was
pointing to Lustre).
- They weren't dirty; they were all marked up-to-date.
From my reading of the Linux VM code, pages marked with private data
is the case here) get their releasepage op called, which in this case is
a pointer to ll_releasepage. I took at look at ll_releasepage, but it wasn't
easy to determine exactly why the page wouldn't be released in this case.
I'm glad to look at anything people could suggest; I was going to instrument
ll_releasepage to see if a) it not releasing pages is the cause of this
problem and b) if it is the problem, to see exactly why it isn't releasing
I took a look at what (I think) was the contents of the pages are, but
it was just binary data, so it wasn't anything helpful to me. I was
thinking that maybe it was directory data (we have one user here who has
directory files that have sizes in the hundreds of megabytes), but it
wasn't obvious to me that it was directory data.