3:01 p.m.
Hi all,
When I run the iozone benchmark on lustre it ends up only using one OSS
server, I assume this has something to do with the way it writes but I'm
not really sure where to look, at the moment I am just using automated
tests...
(iozone -Ra -g 64G)
lustre is functioning fine and when using dd from multiple machines with a
bs=102400 it uses all OSS.
Any pointers would be very welcome...
Thanks,
Eli
3:13 p.m.
When you create a Lustre file system, by default it puts your data on one OST. Check out
section 18.3 (lfs setstripe) in the latest Lustre manual for information on how to change
this for a single file, a single directory tree, or the whole filesystem
(https://wiki.hpdd.intel.com/display/PUB/Documentation).
Shawn
On Feb 20, 2014, at 8:01 AM, E.S. Rosenberg
<esr+hpdd-discuss@mail.hebrew.edu<mailto:esr+hpdd-discuss@mail.hebrew.edu>>
wrote:
Hi all,
When I run the iozone benchmark on lustre it ends up only using one OSS server, I assume
this has something to do with the way it writes but I'm not really sure where to look,
at the moment I am just using automated tests...
(iozone -Ra -g 64G)
lustre is functioning fine and when using dd from multiple machines with a bs=102400 it
uses all OSS.
Any pointers would be very welcome...
Thanks,
Eli
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
3:15 p.m.
Eli,
Your comment about dd means that at least some part of your file system is set up to
stripe across multiple OSTs, but my first guess on hearing this would be that IOZone is
working in a space where the default striping is left at 1. It’s the simplest
explanation, though that doesn’t mean it’s right. :)
- Patrick
From: "E.S. Rosenberg"
<esr+hpdd-discuss@mail.hebrew.edu<mailto:esr+hpdd-discuss@mail.hebrew.edu>>
Date: Thursday, February 20, 2014 at 8:01 AM
To: "hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>"
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: [HPDD-discuss] iozone and lustre
Hi all,
When I run the iozone benchmark on lustre it ends up only using one OSS server, I assume
this has something to do with the way it writes but I'm not really sure where to look,
at the moment I am just using automated tests...
(iozone -Ra -g 64G)
lustre is functioning fine and when using dd from multiple machines with a bs=102400 it
uses all OSS.
Any pointers would be very welcome...
Thanks,
Eli
3:32 p.m.
re:all
On Thu, Feb 20, 2014 at 4:15 PM, Patrick Farrell <paf(a)cray.com> wrote:
Eli,
Your comment about dd means that at least some part of your file system
is set up to stripe across multiple OSTs, but my first guess on hearing
this would be that IOZone is working in a space where the default striping
is left at 1. It's the simplest explanation, though that doesn't mean it's
right. :)
Both dd and iozone are writing into the same directory, my current
assumption is that since iozone constantly overwrites the same file that it
initially created and wrote to in small chunks of data that striping never
steps in...
I assume I can finetune that behavior away....
The lfs operation is run before running a command that will output to a
file a a sort of 'touch' which tells lustre to write a file in a specific
way?
Thanks,
Eli
- Patrick
From: "E.S. Rosenberg" <esr+hpdd-discuss(a)mail.hebrew.edu>
Date: Thursday, February 20, 2014 at 8:01 AM
To: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
Subject: [HPDD-discuss] iozone and lustre
Hi all,
When I run the iozone benchmark on lustre it ends up only using one OSS
server, I assume this has something to do with the way it writes but I'm
not really sure where to look, at the moment I am just using automated
tests...
(iozone -Ra -g 64G)
lustre is functioning fine and when using dd from multiple machines with
a bs=102400 it uses all OSS.
Any pointers would be very welcome...
Thanks,
Eli
3:42 p.m.
lfs setstripe needs to be run before the file is written since striping layout is
determined at file creation time. What you’ll want to do is lfs setstripe -c
<stripe_count> <iozone_output_directory> before you run iozone. Then when you
run iozone, make sure to write to a new file, as a file that already existed will have the
former striping layout. You can use lfs getstripe to verify.
Shawn
On Feb 20, 2014, at 8:32 AM, E.S. Rosenberg
<esr+hpdd-discuss@mail.hebrew.edu<mailto:esr+hpdd-discuss@mail.hebrew.edu>>
wrote:
re:all
On Thu, Feb 20, 2014 at 4:15 PM, Patrick Farrell
<paf@cray.com<mailto:paf@cray.com>> wrote:
Eli,
Your comment about dd means that at least some part of your file system is set up to
stripe across multiple OSTs, but my first guess on hearing this would be that IOZone is
working in a space where the default striping is left at 1. It’s the simplest
explanation, though that doesn’t mean it’s right. :)
Both dd and iozone are writing into the same directory, my current assumption is that
since iozone constantly overwrites the same file that it initially created and wrote to in
small chunks of data that striping never steps in...
I assume I can finetune that behavior away....
The lfs operation is run before running a command that will output to a file a a sort of
'touch' which tells lustre to write a file in a specific way?
Thanks,
Eli
- Patrick
From: "E.S. Rosenberg"
<esr+hpdd-discuss@mail.hebrew.edu<mailto:esr+hpdd-discuss@mail.hebrew.edu>>
Date: Thursday, February 20, 2014 at 8:01 AM
To: "hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>"
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: [HPDD-discuss] iozone and lustre
Hi all,
When I run the iozone benchmark on lustre it ends up only using one OSS server, I assume
this has something to do with the way it writes but I'm not really sure where to look,
at the moment I am just using automated tests...
(iozone -Ra -g 64G)
lustre is functioning fine and when using dd from multiple machines with a bs=102400 it
uses all OSS.
Any pointers would be very welcome...
Thanks,
Eli
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
3:45 p.m.
On Thu, Feb 20, 2014 at 4:42 PM, Hall, Shawn <Shawn.Hall(a)bp.com> wrote:
lfs setstripe needs to be run before the file is written since
striping
layout is determined at file creation time.
I assume it is determined based on initial requested filesize? So if a
program requests a file smaller then stripe size and then ramps up filesize
it will never stripe, correct?
What you'll want to do is lfs setstripe -c <stripe_count>
<iozone_output_directory> before you run iozone. Then when you run iozone,
make sure to write to a new file, as a file that already existed will have
the former striping layout. You can use lfs getstripe to verify.
Shawn
On Feb 20, 2014, at 8:32 AM, E.S. Rosenberg <
esr+hpdd-discuss(a)mail.hebrew.edu> wrote:
re:all
On Thu, Feb 20, 2014 at 4:15 PM, Patrick Farrell <paf(a)cray.com> wrote:
> Eli,
>
> Your comment about dd means that at least some part of your file system
> is set up to stripe across multiple OSTs, but my first guess on hearing
> this would be that IOZone is working in a space where the default striping
> is left at 1. It's the simplest explanation, though that doesn't mean
it's
> right. :)
>
Both dd and iozone are writing into the same directory, my current
assumption is that since iozone constantly overwrites the same file that it
initially created and wrote to in small chunks of data that striping never
steps in...
I assume I can finetune that behavior away....
The lfs operation is run before running a command that will output to a
file a a sort of 'touch' which tells lustre to write a file in a specific
way?
Thanks,
Eli
>
> - Patrick
>
> From: "E.S. Rosenberg" <esr+hpdd-discuss(a)mail.hebrew.edu>
> Date: Thursday, February 20, 2014 at 8:01 AM
> To: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
> Subject: [HPDD-discuss] iozone and lustre
>
> Hi all,
> When I run the iozone benchmark on lustre it ends up only using one OSS
> server, I assume this has something to do with the way it writes but I'm
> not really sure where to look, at the moment I am just using automated
> tests...
>
> (iozone -Ra -g 64G)
>
> lustre is functioning fine and when using dd from multiple machines with
> a bs=102400 it uses all OSS.
>
> Any pointers would be very welcome...
> Thanks,
> Eli
>
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
3:56 p.m.
Not that sophisticated unfortunately. If you have a stripe size of 4, even if you write a
1 byte file it creates 4 objects on 4 different OSTs at 1 MB each.
No matter the file size, the stripe count stays the same. The initial stripe count is
based on 1st if you’ve set the stripe count specifically for that file, 2nd the stripe
count on the directory, 3rd the stripe count on the file system
Shawn
On Feb 20, 2014, at 8:45 AM, E.S. Rosenberg
<esr+hpdd-discuss@mail.hebrew.edu<mailto:esr+hpdd-discuss@mail.hebrew.edu>>
wrote:
On Thu, Feb 20, 2014 at 4:42 PM, Hall, Shawn
<Shawn.Hall@bp.com<mailto:Shawn.Hall@bp.com>> wrote:
lfs setstripe needs to be run before the file is written since striping layout is
determined at file creation time.
I assume it is determined based on initial requested filesize? So if a program requests a
file smaller then stripe size and then ramps up filesize it will never stripe, correct?
What you’ll want to do is lfs setstripe -c <stripe_count>
<iozone_output_directory> before you run iozone. Then when you run iozone, make
sure to write to a new file, as a file that already existed will have the former striping
layout. You can use lfs getstripe to verify.
Shawn
On Feb 20, 2014, at 8:32 AM, E.S. Rosenberg
<esr+hpdd-discuss@mail.hebrew.edu<mailto:esr+hpdd-discuss@mail.hebrew.edu>>
wrote:
re:all
On Thu, Feb 20, 2014 at 4:15 PM, Patrick Farrell
<paf@cray.com<mailto:paf@cray.com>> wrote:
Eli,
Your comment about dd means that at least some part of your file system is set up to
stripe across multiple OSTs, but my first guess on hearing this would be that IOZone is
working in a space where the default striping is left at 1. It’s the simplest
explanation, though that doesn’t mean it’s right. :)
Both dd and iozone are writing into the same directory, my current assumption is that
since iozone constantly overwrites the same file that it initially created and wrote to in
small chunks of data that striping never steps in...
I assume I can finetune that behavior away....
The lfs operation is run before running a command that will output to a file a a sort of
'touch' which tells lustre to write a file in a specific way?
Thanks,
Eli
- Patrick
From: "E.S. Rosenberg"
<esr+hpdd-discuss@mail.hebrew.edu<mailto:esr+hpdd-discuss@mail.hebrew.edu>>
Date: Thursday, February 20, 2014 at 8:01 AM
To: "hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>"
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: [HPDD-discuss] iozone and lustre
Hi all,
When I run the iozone benchmark on lustre it ends up only using one OSS server, I assume
this has something to do with the way it writes but I'm not really sure where to look,
at the moment I am just using automated tests...
(iozone -Ra -g 64G)
lustre is functioning fine and when using dd from multiple machines with a bs=102400 it
uses all OSS.
Any pointers would be very welcome...
Thanks,
Eli
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
5:08 p.m.
On Thu, Feb 20, 2014 at 4:56 PM, Hall, Shawn <Shawn.Hall(a)bp.com> wrote:
Not that sophisticated unfortunately. If you have a stripe size of
4,
even if you write a 1 byte file it creates 4 objects on 4 different OSTs at
1 MB each.
No matter the file size, the stripe count stays the same. The initial
stripe count is based on 1st if you've set the stripe count specifically
for that file, 2nd the stripe count on the directory, 3rd the stripe count
on the file system
I ran more tests and it turns out that my good performace with dd was a
fluke if you will, in that each node was just using an alternate OSS, I
thought it was striping because I was using several nodes at the same time
and seeing both OSS being used, once I ran dd on only one node I saw that
it was actually only writing to a single OSS.
I'm now going to play around with lfs.
Thanks for all the assistance,
Eli
Shawn
On Feb 20, 2014, at 8:45 AM, E.S. Rosenberg <
esr+hpdd-discuss(a)mail.hebrew.edu> wrote:
On Thu, Feb 20, 2014 at 4:42 PM, Hall, Shawn <Shawn.Hall(a)bp.com> wrote:
> lfs setstripe needs to be run before the file is written since striping
> layout is determined at file creation time.
>
I assume it is determined based on initial requested filesize? So if a
program requests a file smaller then stripe size and then ramps up filesize
it will never stripe, correct?
> What you'll want to do is lfs setstripe -c <stripe_count>
> <iozone_output_directory> before you run iozone. Then when you run iozone,
> make sure to write to a new file, as a file that already existed will have
> the former striping layout. You can use lfs getstripe to verify.
>
> Shawn
>
> On Feb 20, 2014, at 8:32 AM, E.S. Rosenberg <
> esr+hpdd-discuss(a)mail.hebrew.edu> wrote:
>
> re:all
>
>
> On Thu, Feb 20, 2014 at 4:15 PM, Patrick Farrell <paf(a)cray.com> wrote:
>
>> Eli,
>>
>> Your comment about dd means that at least some part of your file
>> system is set up to stripe across multiple OSTs, but my first guess on
>> hearing this would be that IOZone is working in a space where the default
>> striping is left at 1. It's the simplest explanation, though that
doesn't
>> mean it's right. :)
>>
> Both dd and iozone are writing into the same directory, my current
> assumption is that since iozone constantly overwrites the same file that it
> initially created and wrote to in small chunks of data that striping never
> steps in...
> I assume I can finetune that behavior away....
>
> The lfs operation is run before running a command that will output to a
> file a a sort of 'touch' which tells lustre to write a file in a specific
> way?
> Thanks,
> Eli
>
>>
>> - Patrick
>>
>> From: "E.S. Rosenberg" <esr+hpdd-discuss(a)mail.hebrew.edu>
>> Date: Thursday, February 20, 2014 at 8:01 AM
>> To: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
>> Subject: [HPDD-discuss] iozone and lustre
>>
>> Hi all,
>> When I run the iozone benchmark on lustre it ends up only using one OSS
>> server, I assume this has something to do with the way it writes but I'm
>> not really sure where to look, at the moment I am just using automated
>> tests...
>>
>> (iozone -Ra -g 64G)
>>
>> lustre is functioning fine and when using dd from multiple machines
>> with a bs=102400 it uses all OSS.
>>
>> Any pointers would be very welcome...
>> Thanks,
>> Eli
>>
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
>
>
>
3011
days inactive
3011
days old
7 comments
3 participants
participants (3)
-
E.S. Rosenberg -
Hall, Shawn -
Patrick Farrell