Re: [HPDD-discuss] MDT keeps in a 100% usage

If you aren't using the ChangeLogs, then you should disable it again: lctl --device jffstg-MDT0000 changelog_deregister That should cause your ChangeLogs to be deleted. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Intel High Performance Data Division On 2016/09/20, 15:46, "Ramiro Alba" <raq(a)cttc.upc.edu&gt; wrote: Hi Andreas, I had enabled changelogs by doing lctl --device jffstg-MDT0000 changelog_register I think it was due to be able to use some recovering funcionalities or rsync related. Not quite sure. Can that be related with my issue? Now I've mounted the MDT using 'ldiskfs' type and done: du -sk CONFIGS O/* oi* I've got 67392548 O/1 => 98% of Used space Inside 0/1 directory there are d0 -> d31 directories of ~2G size each one. What is it the meaning the having all the size in there, then? Related to stripping: cat /proc/fs/lustre/lov/jffstg-MDT0000-mdtlov/stripecount 1 cat /proc/fs/lustre/lov/jffstg-MDT0000-mdtlov/stripesize 1048576 cat /proc/fs/lustre/lov/jffstg-MDT0000-mdtlov/stripeoffset 18446744073709551615 Any suggestion from this info? Regards On 2016-09-20 14:55, Dilger, Andreas wrote: If you don't have a ChangeLog consumer then the records will continue to be added to the log, but nothing will ever be removed from the log and it will grow to fill all available space on the MDT. You could try mounting the MDT locally as type ldiskfs and run "du -sk CONFIGS O/* oi*" to see if any of those directories or files are consuming a lot of space, which would help determine what is the source of the problem. The other possibility is if you are using a default stripe count of 6 or more and your MDT wasn't formatted to take this into account. In that case, each file on the MDT would consume an extra 4KB block for the file layout, which can take more space on the MDT than expected. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Intel High Performance Data Division On 2016/09/20, 12:46, "HPDD-discuss on behalf of Ramiro Alba" <hpdd-discuss-bounces(a)lists.01.org on behalf of raq(a)cttc.upc.edu&gt; wrote: Yes. Is it related? Regards El 20 de septiembre de 2016 12:10:29 CEST, Colin Faber <cfaber(a)gmail.com&gt; escribió: Are you running with changelogs enabled by chance? On Sep 20, 2016 12:07 PM, "Ramiro Alba" <raq(a)cttc.upc.edu&gt; wrote: Hi all, I am using Lustre 2.5.3 under CentOS 6.5 both servers and clients and after removing 300.000 files (from 1.600.000 files), only 163008Kb could be reduced on MDT, but it keeps showing a 100% usage: Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vglustre-MDT 70766580 68563552 0 100% /lustre/mdt The lustre file system is using 6 OSTs of 30TB size The system mounts without problems, MDT backups (using tar) can be done and the whole file system seems to work fine, but I can not manage to reduce the MDT usage. The only thing is that yesterday the whole cluster system, due to an electrical problem, went down suddenly, without been protected by our UPS. Any chance to be related with keeping on 100% usage? I am worried with this issue. Should I? Apart from using a bigger disk on MDT, is here anything that can be done? Any suggestions are welcomed Best regards ############################################################# I did: [root@jffmds check]# e2fsck -fp /dev/mapper/vglustre-MDT [QUOTA WARNING] Usage inconsistent for ID 0:actual (69414895616, 21842) != expected (69394186240, 21839) [QUOTA WARNING] Usage inconsistent for ID 1056:actual (1572864, 6056) != expected (1572864, 6047) [QUOTA WARNING] Usage inconsistent for ID 1027:actual (9265152, 21891) != expected (9265152, 21795) [QUOTA WARNING] Usage inconsistent for ID 1367:actual (40321024, 121180) != expected (40316928, 121054) [QUOTA WARNING] Usage inconsistent for ID 1304:actual (6074368, 32508) != expected (6074368, 32507) [QUOTA WARNING] Usage inconsistent for ID 1041:actual (9113600, 28693) != expected (9113600, 28687) [QUOTA WARNING] Usage inconsistent for ID 1031:actual (5332992, 33716) != expected (5320704, 33578) [QUOTA WARNING] Usage inconsistent for ID 1037:actual (1925120, 8141) != expected (1925120, 8138) jffstg-MDT0000: Update quota info for quota type 0. [QUOTA WARNING] Usage inconsistent for ID 0:actual (69414907904, 21870) != expected (69394198528, 21867) [QUOTA WARNING] Usage inconsistent for ID 100:actual (296939520, 1240117) != expected (296923136, 1239738) jffstg-MDT0000: Update quota info for quota type 1. jffstg-MDT0000: 1262302/47185920 files (1.1% non-contiguous), 23038694/23592960 blocks but it keeps saying 100% usage -- Ramiro Alba Centre Tecnològic de Tranferència de Calor http://www.cttc.upc.edu Escola Tècnica Superior d'Enginyeries Industrial i Aeronàutica de Terrassa Colom 11, E-08222, Terrassa, Barcelona, Spain Tel: (+34) 93 739 8928 [1] -- Aquest missatge ha estat analitzat per MAILSCANNER [2] a la cerca de virus i d'altres continguts perillosos, i es considera que está net. _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss -- Aquest missatge ha estat analitzat per MAILSCANNER [2] a la cerca de virus i d'altres continguts perillosos, i es considera que está net.