OK!  Well aim for a larger size test and see how it goes.  Thanks for the responses.

Mb

On Apr 11, 2014 4:42 PM, "Dilger, Andreas" <andreas.dilger@intel.com> wrote:
On 2014/04/11, 1:22 PM, "Matt Bettinger" <iamatt@gmail.com> wrote:

>Finally built 2.4.1 client on sles 11.2 and testing against 2.4.3 file
>system.  We still get major performance problems when writing to quota
>directory .  Reads are Ok but writes of a user writing a a directory
>with a quota for his userid  drop from 364 MB/s to 5-7 MB/s
>
>root or someone else can write to the same directory with the expected
>speeds ~360-70 MB/s (even though gateway router boxes)  Any
>suggestions in trying to see what would be going on?

If you are close to running out of quota, Lustre has to throttle the IO so
that it doesn't overshoot the quota limits significantly.  In your
example, the process has exceeded quota with only 300MB.  Typically,
Lustre users will have many GB or TB of quota. If there is a lot of quota
available then it can go faster since the client isn't in danger of
exceeding the limit.

Cheers, Andreas

>Here is what our mdt and some of the ost are showing when quota is
>enabled..
>
>[root@lfs-mds-0-0 ~]#  lctl get_param osd-*.*.quota_slave.info
>osd-ldiskfs.lustre2-MDT0000.quota_slave.info=
>target name:    lustre2-MDT0000
>pool ID:        0
>type:           md
>quota enabled:  ug
>conn to master: setup
>space acct:     ug
>user uptodate:  glb[1],slv[1],reint[0]
>group uptodate: glb[1],slv[1],reint[0]
>[root@lfs-mds-0-0 ~]# ssh lfs-oss-0-0
>Last login: Fri Apr 11 11:40:43 2014 from 172.20.0.1
>
>
>[root@lfs-oss-0-0 ~]#  lctl get_param osd-*.*.quota_slave.info
>osd-ldiskfs.lustre2-OST0000.quota_slave.info=
>target name:    lustre2-OST0000
>pool ID:        0
>type:           dt
>quota enabled:  ug
>conn to master: setup
>space acct:     ug
>user uptodate:  glb[1],slv[1],reint[0]
>group uptodate: glb[1],slv[1],reint[0]
>osd-ldiskfs.lustre2-OST0001.quota_slave.info=
>target name:    lustre2-OST0001
>pool ID:        0
>type:           dt
>quota enabled:  ug
>conn to master: setup
>space acct:     ug
>user uptodate:  glb[1],slv[1],reint[0]
>group uptodate: glb[1],slv[1],reint[0]
>osd-ldiskfs.lustre2-OST0002.quota_slave.info=
>target name:    lustre2-OST0002
>pool ID:        0
>type:           dt
>quota enabled:  ug
>conn to master: setup
>space acct:     ug
>user uptodate:  glb[1],slv[1],reint[0]
>group uptodate: glb[1],slv[1],reint[0]
>
>
>Quota works just VERY slow..
>
>l0363734@hous0162:/lustre2/mattb/300MB_QUOTA> dd if=/dev/zero
>of=big.file3 bs=100M count=1000
>dd: writing `big.file3': Disk quota exceeded
>3+0 records in
>2+0 records out
>314507264 bytes (315 MB) copied, 44.3913 s, 7.1 MB/s
>
>^_^
>
>On Wed, Apr 9, 2014 at 12:24 PM, Patrick Farrell <paf@cray.com> wrote:
>> Matt,
>>
>> We at Cray never specifically performance tested 1.8.x clients to 2.4.x
>> servers with quotas enabled, but I can say that quotas on 2.4 servers
>>with
>> 2.4 clients showed almost no measurable performance difference (with
>> quotas vs without quotas).
>>
>> In your situation, I¹d suggest, if possible, trying a single 2.4 client
>> and seeing if the performance is good.  That should tell you whether the
>> problem is with 1.8.x clients to 2.4 servers or if it's something else
>>on
>> your system.
>>
>> - Patrick
>>
>> On 4/9/14, 12:54 PM, "Matt Bettinger" <iamatt@gmail.com> wrote:
>>
>>>Hello,
>>>
>>>We have a fresh 2.4.3 lustre upgrade that is not yet put into
>>>production running on rhel 6.4.
>>>
>>>We would like to take a look at quotas but looks like there is some
>>>major performance problems with 1.8.9 clients.
>>>
>>>Here is how I enabled quotas
>>>
>>>[root@lfs-mds-0-0 ~]# lctl conf_param lustre2.quota.mdt=ug
>>>[root@lfs-mds-0-0 ~]# lctl conf_param lustre2.quota.ost=ug
>>>
>>>
>>>
>>>[root@lfs-mds-0-0 ~]#  lctl get_param osd-*.*.quota_slave.info
>>>osd-ldiskfs.lustre2-MDT0000.quota_slave.info=
>>>target name:    lustre2-MDT0000
>>>pool ID:        0
>>>type:           md
>>>quota enabled:  ug
>>>conn to master: setup
>>>space acct:     ug
>>>user uptodate:  glb[1],slv[1],reint[0]
>>>group uptodate: glb[1],slv[1],reint[0]
>>>
>>>The quotas seem to be working however the write performance from
>>>1.8.9wc client to 2.4.3 with quotas on is horrific.  Am I not setting
>>>quotas up correctly?
>>>
>>>
>>>I try to make a simple user quota on /lustre2/mattb/300MB_QUOTA
>>>directory
>>>
>>>[root@hous0036 mattb]# lfs setquota -u l0363734 -b 307200 -B 309200 -i
>>>10000 -I 11000 /lustre2/mattb/300MB_QUOTA/
>>>
>>>See quota change is in effect...
>>>
>>>[root@hous0036 mattb]# lfs quota -u l0363734 /lustre2/mattb/300MB_QUOTA/
>>>Disk quotas for user l0363734 (uid 1378):
>>>     Filesystem  kbytes   quota   limit   grace   files   quota   limit
>>>grace
>>>/lustre2/mattb/300MB_QUOTA/
>>>                 310292* 307200  309200       -       4   10000   11000
>>>    -
>>>
>>>
>>>Try and write to quota directory as the user but get horrible write
>>>speed
>>>
>>>[l0363734@hous0036 300MB_QUOTA]$ dd if=/dev/zero of=301MB_FILE bs=1M
>>>count=301
>>>301+0 records in
>>>301+0 records out
>>>315621376 bytes (316 MB) copied, 61.7426 seconds, 5.1 MB/s
>>>
>>>Try file number 2 and then quota take effect, so it seems.
>>>
>>>[l0363734@hous0036 300MB_QUOTA]$ dd if=/dev/zero of=301MB_FILE2 bs=1M
>>>count=301
>>>dd: writing `301MB_FILE2': Disk quota exceeded
>>>dd: closing output file `301MB_FILE2': Input/output error
>>>
>>>
>>>If I disable quotas using
>>>
>>>[root@lfs-mds-0-0 ~]# lctl conf_param lustre2.quota.mdt=none
>>>[root@lfs-mds-0-0 ~]# lctl conf_param lustre2.quota.oss=none
>>>
>>>
>>>Then try and write the same file the speeds are more like we expect
>>>but then can't use quotas.
>>>
>>>[l0363734@hous0036 300MB_QUOTA]$ dd if=/dev/zero of=301MB_FILE2 bs=1M
>>>count=301
>>>301+0 records in
>>>301+0 records out
>>>315621376 bytes (316 MB) copied, 0.965009 seconds, 327 MB/s
>>>[l0363734@hous0036 300MB_QUOTA]$ dd if=/dev/zero of=301MB_FILE2 bs=1M
>>>count=301
>>>
>>>
>>>I have not tried this with a 2.4 client, yet since all of our nodes
>>>are 1.8.X until we rebuild our images.
>>>
>>>  I was going by the manual on
>>>http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifac
>>>t/
>>>lustre_manual.xhtml#quota_administration
>>> but it looks like I am running into interoperability issue (which I
>>>thought I fixed by using 1.8.9-wc client)  or just not configuring
>>>this correctly.
>>>
>>>
>>>Thanks!
>>>
>>>MB
>>>_______________________________________________
>>>HPDD-discuss mailing list
>>>HPDD-discuss@lists.01.org
>>>https://lists.01.org/mailman/listinfo/hpdd-discuss
>>
>_______________________________________________
>HPDD-discuss mailing list
>HPDD-discuss@lists.01.org
>https://lists.01.org/mailman/listinfo/hpdd-discuss
>


Cheers, Andreas
--
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division