If you don't have a ChangeLog consumer then the records will continue to be added to the log, but nothing will ever be removed from the log and it will grow to fill all available space on the MDT.

 

You could try mounting the MDT locally as type ldiskfs and run "du -sk CONFIGS O/* oi*" to see if any of those directories or files are consuming a lot of space, which would help determine what is the source of the problem.

 

The other possibility is if you are using a default stripe count of 6 or more and your MDT wasn't formatted to take this into account.  In that case, each file on the MDT would consume an extra 4KB block for the file layout, which can take more space on the MDT than expected.

 

Cheers, Andreas

-- 

Andreas Dilger

Lustre Principal Architect

Intel High Performance Data Division

 

On 2016/09/20, 12:46, "HPDD-discuss on behalf of Ramiro Alba" <hpdd-discuss-bounces@lists.01.org on behalf of raq@cttc.upc.edu> wrote:

 

Yes. Is it related?

Regards

El 20 de septiembre de 2016 12:10:29 CEST, Colin Faber <cfaber@gmail.com> escribió:

Are you running with changelogs enabled by chance?

 

On Sep 20, 2016 12:07 PM, "Ramiro Alba" <raq@cttc.upc.edu> wrote:

Hi all,

I am using Lustre 2.5.3 under CentOS 6.5 both servers and clients and

after removing 300.000 files (from 1.600.000 files), only 163008Kb could

be reduced on MDT, but it keeps showing a 100% usage:

Filesystem 1K-blocks   Used        Available Use% Mounted on
/dev/mapper/vglustre-MDT
                   70766580  68563552 0             100% /lustre/mdt

The lustre file system is using 6 OSTs of 30TB size
 

 

The system mounts without problems, MDT backups (using tar) can be done

and the whole file system seems to work fine, but I can not manage to reduce

the MDT usage. The only thing is that yesterday the whole cluster system, due

to an electrical problem, went down suddenly, without been protected by our UPS.

Any chance to be related with keeping on 100% usage?

I am worried with this issue. Should I?

Apart from using a bigger disk on MDT, is here anything that can be done?

Any suggestions are welcomed

Best regards

#############################################################

I did:

[root@jffmds check]# e2fsck -fp /dev/mapper/vglustre-MDT
[QUOTA WARNING] Usage inconsistent for ID 0:actual (69414895616, 21842) != expected (69394186240, 21839)
[QUOTA WARNING] Usage inconsistent for ID 1056:actual (1572864, 6056) != expected (1572864, 6047)
[QUOTA WARNING] Usage inconsistent for ID 1027:actual (9265152, 21891) != expected (9265152, 21795)
[QUOTA WARNING] Usage inconsistent for ID 1367:actual (40321024, 121180) != expected (40316928, 121054)
[QUOTA WARNING] Usage inconsistent for ID 1304:actual (6074368, 32508) != expected (6074368, 32507)
[QUOTA WARNING] Usage inconsistent for ID 1041:actual (9113600, 28693) != expected (9113600, 28687)
[QUOTA WARNING] Usage inconsistent for ID 1031:actual (5332992, 33716) != expected (5320704, 33578)
[QUOTA WARNING] Usage inconsistent for ID 1037:actual (1925120, 8141) != expected (1925120, 8138)
jffstg-MDT0000: Update quota info for quota type 0.
[QUOTA WARNING] Usage inconsistent for ID 0:actual (69414907904, 21870) != expected (69394198528, 21867)
[QUOTA WARNING] Usage inconsistent for ID 100:actual (296939520, 1240117) != expected (296923136, 1239738)
jffstg-MDT0000: Update quota info for quota type 1.
jffstg-MDT0000: 1262302/47185920 files (1.1% non-contiguous), 23038694/23592960 blocks

but it keeps saying 100% usage

--

Ramiro Alba

Centre Tecnològic de Tranferència de Calor

http://www.cttc.upc.edu

Escola Tècnica Superior d'Enginyeries

Industrial i Aeronàutica de Terrassa
Colom 11, E-08222, Terrassa, Barcelona, Spain
Tel: (+34) 93 739 8928


--
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que está net.


_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss


--
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que está net.


--
Ramiro Alba

Centre Tecnològic de Tranferència de Calor
http://www.cttc.upc.edu

Escola Tècnica Superior d'Enginyeries
Industrial i Aeronàutica de Terrassa
Colom 11, E-08222, Terrassa, Barcelona, Spain
Tel: (+34) 93 739 8928
--
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que está net.