5:36 a.m.
Dear All,
Running into tons of "Unattached inode ..,Connect to /lost+found ..., Inode
<....> ref count is 2, should be 1" on MDT, after running lfs_migrate for
entire last month trying to move files off of old storage hardware. Lustre v.2.4.3
I ran e2fsck -fp on OST's and it worked fine, since they had only couple of such
messages they quickly got fixed.
Ran e2fsck -fp on MDT and it fails to fix it, as it is not prudent enough. Only option
left was to run it with "e2fsck -fy". In doing so i see tons of above messages,
that are being fixed. I realized I had orphaned objects, but I did not think that I would
find never ending objects to fix. I have been running "e2fsck -fy MDT" all today
and NO luck or sign on completion.
With this pace, I am at 82M inodes in fixing ...based on the output of my e2fsck -fy MDT
run. Given 2TB MDT, I should be capable of at least 512M inodes. I have a question in this
context.
Q) My file system had about 35 million files(guess) and MDT usage was about 32GB, any idea
how long my e2fsck checks could take?
Q) Why am I seeing these many Unattached inodes in such large numbers, is there something
bad going on with our file system?
Please advise.
Thank you,
Amit
10:05 a.m.
This is a bug in older versions of Lustre during lfs migrate that has been fixed. The
inodes going into lost+found are the temporary inodes used during migration and are AFAIK
not useful for anything and could be deleted, though I couldn't say with 100%
certainty of your case may be different. While this hit a few sites, most people are
running 2.5.x these days, and they didn't migrate so many files, so it hasn't been
a big problem to date. Upgrading at this point will not fix those inodes, only prevent new
ones from appearing.
I suspect the reason the e2fsck is so slow is because lost+found must be getting huge. You
could kill the e2fsck, mount the MDT as ldiskfs, and clear out those inodes after either
by deleting them (if you are sure that your "good" files are still intact), or
by moving the whole lost+found directory into /ROOT and creating a new one. That said, the
lost+found directory might only be useful up to ~1M files before it bogs down under O(n^2)
insertions.
If you need to get your system back online, these orphans are not harmful (besides using
up your inodes), so you could clean them up at a later time.
Cheers, Andreas
On Jun 28, 2016, at 21:36, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
Dear All,
Running into tons of "Unattached inode ..,Connect to /lost+found ..., Inode
<....> ref count is 2, should be 1" on MDT, after running lfs_migrate for
entire last month trying to move files off of old storage hardware. Lustre v.2.4.3
I ran e2fsck -fp on OST's and it worked fine, since they had only couple of such
messages they quickly got fixed.
Ran e2fsck -fp on MDT and it fails to fix it, as it is not prudent enough. Only option
left was to run it with "e2fsck -fy". In doing so i see tons of above messages,
that are being fixed. I realized I had orphaned objects, but I did not think that I would
find never ending objects to fix. I have been running "e2fsck -fy MDT" all today
and NO luck or sign on completion.
With this pace, I am at 82M inodes in fixing ...based on the output of my e2fsck -fy MDT
run. Given 2TB MDT, I should be capable of at least 512M inodes. I have a question in this
context.
Q) My file system had about 35 million files(guess) and MDT usage was about 32GB, any
idea how long my e2fsck checks could take?
Q) Why am I seeing these many Unattached inodes in such large numbers, is there something
bad going on with our file system?
Please advise.
Thank you,
Amit
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
8:42 p.m.
Hi Andreas,
Thank you for your insight. I resorted to canceling e2fsck and bring up the file system as
is. Hope to get off this system sooner.
Regards,
Amit
-----Original Message-----
From: Dilger, Andreas [mailto:andreas.dilger@intel.com]
Sent: Wednesday, June 29, 2016 3:05 AM
To: Kumar, Amit <ahkumar(a)mail.smu.edu>
Cc: <hpdd-discuss(a)lists.01.org> <hpdd-discuss(a)lists.01.org>
Subject: Re: [HPDD-discuss] Unattached inode, Connect to /lost+found, Inode ref count is
2, should be 1
This is a bug in older versions of Lustre during lfs migrate that has been fixed. The
inodes going into lost+found are the temporary inodes used during migration and are AFAIK
not useful for anything and could be deleted, though I couldn't say with 100%
certainty of your case may be different. While this hit a few sites, most people are
running 2.5.x these days, and they didn't migrate so many files, so it hasn't been
a big problem to date. Upgrading at this point will not fix those inodes, only prevent new
ones from appearing.
I suspect the reason the e2fsck is so slow is because lost+found must be getting huge. You
could kill the e2fsck, mount the MDT as ldiskfs, and clear out those inodes after either
by deleting them (if you are sure that your "good" files are still intact), or
by moving the whole lost+found directory into /ROOT and creating a new one. That said, the
lost+found directory might only be useful up to ~1M files before it bogs down under O(n^2)
insertions.
If you need to get your system back online, these orphans are not harmful (besides using
up your inodes), so you could clean them up at a later time.
Cheers, Andreas
On Jun 28, 2016, at 21:36, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
Dear All,
Running into tons of "Unattached inode ..,Connect to /lost+found ..., Inode
<....> ref count is 2, should be 1" on MDT, after running lfs_migrate for
entire last month trying to move files off of old storage hardware. Lustre v.2.4.3
I ran e2fsck -fp on OST's and it worked fine, since they had only couple of such
messages they quickly got fixed.
Ran e2fsck -fp on MDT and it fails to fix it, as it is not prudent enough. Only option
left was to run it with "e2fsck -fy". In doing so i see tons of above messages,
that are being fixed. I realized I had orphaned objects, but I did not think that I would
find never ending objects to fix. I have been running "e2fsck -fy MDT" all today
and NO luck or sign on completion.
With this pace, I am at 82M inodes in fixing ...based on the output of my e2fsck -fy MDT
run. Given 2TB MDT, I should be capable of at least 512M inodes. I have a question in this
context.
Q) My file system had about 35 million files(guess) and MDT usage was about 32GB, any
idea how long my e2fsck checks could take?
Q) Why am I seeing these many Unattached inodes in such large numbers, is there something
bad going on with our file system?
Please advise.
Thank you,
Amit
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
1978
days inactive
1978
days old
2 comments
2 participants
participants (2)
-
Dilger, Andreas -
Kumar, Amit