8:28 a.m.
Hello everybody.
Here a strange problem that maybe could be reported as a bug.
Explanation:
Lustre 2.7 (Centos 6.6) in MDS, 2OSS's (12 OST's) and ~200 Clients. 1
single Lustre Filesystem.
Execution of "ungrib.exe
<http://www2.mmm.ucar.edu/wrf/OnLineTutorial/Basics/UNGRIB/index.html>&...
for WRF into any directory with "stripe = 1" works perfectly.
Execution of "ungrib.exe" for WRF into any directory with any "stripe
<>
1", fails.
"ungrib.exe" works writting medium-size (610MB) files (34 in my case) in
the working directory. If the working directory is configured with
"stripe = 1" (each file goes to a single OST), the proccess makes its
job and finish cleanly.
If the working directory is configured with any other stripe (probed
with 2, 4, 8, 12 and -1), when writing one of the 610MB files (not
always the same one), the procces aborts.
Executing with "strace", the last lines of the trace are:
/[...]
//lseek(4, 99309154, SEEK_SET) = 99309154//
//read(4,
"GRIB\0\0\0\2\0\0\0\0\0\v5\304\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
512) = 512//
//lseek(4, 100043810, SEEK_SET) = 100043810//
//read(4, "7777", 4) = 4//
//lseek(4, 99309154, SEEK_SET) = 99309154//
//read(4,
"GRIB\0\0\0\2\0\0\0\0\0\v5\304\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
734660) = 734660//
//brk(0x1e41f000) = 0x1e41f000//
//lseek(4, 100043814, SEEK_SET) = 100043814//
//read(4,
"GRIB\0\0\0\2\0\0\0\0\0\22:\376\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
512) = 512//
//lseek(4, 101238560, SEEK_SET) = 101238560//
//read(4, "7777", 4) = 4//
//lseek(4, 100043814, SEEK_SET) = 100043814//
//read(4,
"GRIB\0\0\0\2\0\0\0\0\0\22:\376\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
1194750) = 619482//
//write(3, "2016-04-19 17:15:43.713 --- ", 28) = 28//
//write(1, "ERROR: ", 7ERROR: ) = 7//
//write(3, "ERROR: ", 7) = 7//
//write(1, "rd_grib2: IO Error. ", 20rd_grib2: IO Error. ) = 20//
//write(3, "rd_grib2: IO Error. ", 20) = 20//
//write(1, "1194750", 71194750) = 7//
//write(3, "1194750", 7) = 7//
//write(1, " .ne. ", 6 .ne. ) = 6//
//write(3, " .ne. ", 6) = 6//
//write(1, "619482", 6619482) = 6//
//write(3, "619482", 6) = 6//
//write(1, "\n", 1//
//) = 1//
//write(3, "\n", 1) = 1//
//exit_group(0) = ?/
All OST's look to be OK. no timeouts are detected. OSS's and MDS are not
reporting errors. The only pattern detected is the stripe of the working
directory where data is written. There's no a specific "breaking point"
of the proccess. One time it can occur writing a file and other time it
can occur writing a different one.
This issue has been detected with other executions that write
medium-size files, but no way to reproduce it easily.
Any idea of what can be happening?.
Thank you so much.
--
*Jose Manuel Martínez García / Tel. 987 293 174 *
*Coordinador de Sistemas*
Fundación Centro de Supercomputación de Castilla y León
Edificio CRAI-TIC, Campus de Vegazana, s/n
Universidad de León
24071 León, España
www.fcsc.es
logoFCSCL jcyl
_________________________________
Este correo va dirigido, de manera exclusiva, a su destinatario y puede
contener información confidencial, cuya divulgación no está permitida
por la ley. Si usted no es su destinatario notifíquelo urgentemente al
remitente y borre este correo de su sistema.
Proteja el Medio Ambiente. Evite imprimir este mensaje si no es
estrictamente necesario.
8:44 a.m.
When the stripe count is greater than 1 and you see these errors, have you captured the
striping info from the file that gives the errors? I’m wondering if they might have an
OST in common that could point to a problem. Likewise, when you use stripe=1, have you
checked the striping info to see if all the OSTs were used for at least one of the files?
(If so, that could be used to rule out problems with the OSTs.)
Can you tell us anything about the underlying hardware that is used? Are you using
storage controllers or just internal drives?
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
On Apr 19, 2016, at 11:28 AM, Jose Manuel Martínez
<jose.martinez(a)fcsc.es> wrote:
Hello everybody.
Here a strange problem that maybe could be reported as a bug.
Explanation:
Lustre 2.7 (Centos 6.6) in MDS, 2OSS's (12 OST's) and ~200 Clients. 1 single
Lustre Filesystem.
Execution of "ungrib.exe" for WRF into any directory with "stripe =
1" works perfectly.
Execution of "ungrib.exe" for WRF into any directory with any "stripe
<> 1", fails.
"ungrib.exe" works writting medium-size (610MB) files (34 in my case) in the
working directory. If the working directory is configured with "stripe = 1"
(each file goes to a single OST), the proccess makes its job and finish cleanly.
If the working directory is configured with any other stripe (probed with 2, 4, 8, 12 and
-1), when writing one of the 610MB files (not always the same one), the procces aborts.
Executing with "strace", the last lines of the trace are:
[...]
lseek(4, 99309154, SEEK_SET) = 99309154
read(4, "GRIB\0\0\0\2\0\0\0\0\0\v5\304\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
512) = 512
lseek(4, 100043810, SEEK_SET) = 100043810
read(4, "7777", 4) = 4
lseek(4, 99309154, SEEK_SET) = 99309154
read(4, "GRIB\0\0\0\2\0\0\0\0\0\v5\304\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
734660) = 734660
brk(0x1e41f000) = 0x1e41f000
lseek(4, 100043814, SEEK_SET) = 100043814
read(4,
"GRIB\0\0\0\2\0\0\0\0\0\22:\376\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"..., 512) =
512
lseek(4, 101238560, SEEK_SET) = 101238560
read(4, "7777", 4) = 4
lseek(4, 100043814, SEEK_SET) = 100043814
read(4,
"GRIB\0\0\0\2\0\0\0\0\0\22:\376\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
1194750) = 619482
write(3, "2016-04-19 17:15:43.713 --- ", 28) = 28
write(1, "ERROR: ", 7ERROR: ) = 7
write(3, "ERROR: ", 7) = 7
write(1, "rd_grib2: IO Error. ", 20rd_grib2: IO Error. ) = 20
write(3, "rd_grib2: IO Error. ", 20) = 20
write(1, "1194750", 71194750) = 7
write(3, "1194750", 7) = 7
write(1, " .ne. ", 6 .ne. ) = 6
write(3, " .ne. ", 6) = 6
write(1, "619482", 6619482) = 6
write(3, "619482", 6) = 6
write(1, "\n", 1
) = 1
write(3, "\n", 1) = 1
exit_group(0) = ?
All OST's look to be OK. no timeouts are detected. OSS's and MDS are not
reporting errors. The only pattern detected is the stripe of the working directory where
data is written. There's no a specific "breaking point" of the proccess. One
time it can occur writing a file and other time it can occur writing a different one.
This issue has been detected with other executions that write medium-size files, but no
way to reproduce it easily.
Any idea of what can be happening?.
Thank you so much.
--
JoseManuel Martínez García / Tel. 987 293 174
Coordinadorde Sistemas
FundaciónCentro de Supercomputación de Castilla y León
EdificioCRAI-TIC, Campus de Vegazana, s/n
Universidadde León
24071León, España
www.fcsc.es
<ehegidja.png> <edffeedi.png>
_________________________________
Este puede contener informac
Proteja el Medio Ambiente. Evite imprimir este mensaje si no es estrictamente
necesario.
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
9:33 a.m.
This sounds exactly like https://jira.hpdd.intel.com/browse/LU-6389
"read()/write() returning less than available bytes intermittently" which
was fixed after 2.7.0 was released.
There is a patch for Lustre clients to fix this behaviour, but it is not
strictly a bug in Lustre either - this behaviour is allowed by POSIX. I'd
also recommend for you to send a patch to the application so that it will
retry file reads that return less than the full amount of data requested.
Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel High Performance Data Division
On 2016/04/19, 11:28, "Jose Manuel Martínez" <djose.martinez(a)fcsc.es>
wrote:
Hello everybody.
Here a strange problem that maybe could be reported as a bug.
Explanation:
Lustre 2.7 (Centos 6.6) in MDS, 2OSS's (12 OST's) and ~200 Clients. 1
single Lustre Filesystem.
Execution of "ungrib.exe
<http://www2.mmm.ucar.edu/wrf/OnLineTutorial/Basics/UNGRIB/index.html>&...
for WRF into any directory with "stripe = 1" works perfectly.
Execution of "ungrib.exe" for WRF into any directory with any "stripe
<>
1", fails.
"ungrib.exe" works writting medium-size (610MB) files (34 in my case) in
the working directory. If the working directory is configured with
"stripe = 1" (each file goes to a single OST), the proccess makes its job
and finish cleanly.
If the working directory is configured with any other stripe (probed with
2, 4, 8, 12 and -1), when writing one of the 610MB files (not always the
same one), the procces aborts.
Executing with "strace", the last lines of the trace are:
[...]
lseek(4, 99309154, SEEK_SET) = 99309154
read(4,
"GRIB\0\0\0\2\0\0\0\0\0\v5\304\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
512) = 512
lseek(4, 100043810, SEEK_SET) = 100043810
read(4, "7777", 4) = 4
lseek(4, 99309154, SEEK_SET) = 99309154
read(4,
"GRIB\0\0\0\2\0\0\0\0\0\v5\304\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
734660) = 734660
brk(0x1e41f000) = 0x1e41f000
lseek(4, 100043814, SEEK_SET) = 100043814
read(4,
"GRIB\0\0\0\2\0\0\0\0\0\22:\376\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
512) = 512
lseek(4, 101238560, SEEK_SET) = 101238560
read(4, "7777", 4) = 4
lseek(4, 100043814, SEEK_SET) = 100043814
read(4,
"GRIB\0\0\0\2\0\0\0\0\0\22:\376\0\0\0\25\1\0\7\0\0\2\1\1\7\340\4\22"...,
1194750) = 619482
write(3, "2016-04-19 17:15:43.713 --- ", 28) = 28
write(1, "ERROR: ", 7ERROR: ) = 7
write(3, "ERROR: ", 7) = 7
write(1, "rd_grib2: IO Error. ", 20rd_grib2: IO Error. ) = 20
write(3, "rd_grib2: IO Error. ", 20) = 20
write(1, "1194750", 71194750) = 7
write(3, "1194750", 7) = 7
write(1, " .ne. ", 6 .ne. ) = 6
write(3, " .ne. ", 6) = 6
write(1, "619482", 6619482) = 6
write(3, "619482", 6) = 6
write(1, "\n", 1
) = 1
write(3, "\n", 1) = 1
exit_group(0) = ?
All OST's look to be OK. no timeouts are detected. OSS's and MDS are not
reporting errors. The only pattern detected is the stripe of the working
directory where data is written. There's no a specific "breaking point"
of the proccess. One time it can occur
writing a file and other time it can occur writing a different one.
This issue has been detected with other executions that write medium-size
files, but no way to reproduce it easily.
Any idea of what can be happening?.
Thank you so much.
--
Jose Manuel Martínez García / Tel. 987 293 174
Coordinador de Sistemas
Fundación Centro de Supercomputación de Castilla y León
Edificio CRAI-TIC, Campus de Vegazana, s/n
Universidad de León
24071 León, España
<www.fcsc.es>www.fcsc.es <http://www.fcsc.es>
_________________________________
Este correo va dirigido, de manera exclusiva, a su destinatario y puede
contener información confidencial, cuya divulgación no está permitida por
la ley. Si usted no es su destinatario
notifíquelo urgentemente al remitente y borre este correo de su sistema.
Proteja el Medio Ambiente. Evite imprimir este mensaje si no es
estrictamente necesario.
1:04 a.m.
You hit the nail!!.
I had seen the related https://jira.hpdd.intel.com/browse/LU-6392 but I
did not found the relationship with our issue.
After some research, LU-6389 fits exactlly our problems: ungrib.exe is
also written in Fortran and the 'corrupted' stripped files are
reporting 2 or 3 MB less than they should.
After upgrading a pair of Lustre clients to 2.8, the problem has gone.
We will contact WRF developers in order to help patching ungrib.exe.
Thank you so much.
Best regards, JoseM
El 19/04/16 a las 18:33, Dilger, Andreas escribió:
This sounds exactly like https://jira.hpdd.intel.com/browse/LU-6389
"read()/write() returning less than available bytes intermittently" which
was fixed after 2.7.0 was released.
There is a patch for Lustre clients to fix this behaviour, but it is not
strictly a bug in Lustre either - this behaviour is allowed by POSIX. I'd
also recommend for you to send a patch to the application so that it will
retry file reads that return less than the full amount of data requested.
Cheers, Andreas
2305
days inactive
2306
days old
3 comments
3 participants
participants (3)
-
Dilger, Andreas -
Jose Manuel Martínez -
Mohr Jr, Richard Frank (Rick Mohr)