10:55 p.m.
We have also got significant performance improvement with 4MB RPCs (with
Lustre 2.6.53) and increasing max_rpcs_in_flight
However, the more importantly, you should have end to end 4MB IO from
client to disk.
I mean clients send to server with 4MB RPCs, but OSS also needs to
pass
efficient IO size to OSTs. I believe you are missing this part.
I have a question here,
Our OST is built over md/raid6 device. Even if I can see in brw_stats that a
RPC req. is 4MB, it is getting fragmented into 1M requests. My md/raid6 is
receiving 1M writes.
I was looking at following from https://jira.hpdd.intel.com/browse/LU-2598
<snip>
The http://review.whamcloud.com/4993 patch looks like it will resolve this
problem in osd-ldiskfs/osd-io.c:
- bio = bio_alloc(GFP_NOIO, max(BIO_MAX_PAGES,
+ bio = bio_alloc(GFP_NOIO, min(BIO_MAX_PAGES,
</snip>
BIO_MAX_PAGES is defined as 256 (even in latest vanilla kernel).
With above change (max -> min) bio will never get allocated more than 256
pages i.e. 1MB?
Am I understanding it correctly?
Is there any other kernel patch or lustre patch which changes BIO_MAX_PAGES
or rectifies this issue, if any?
Regarding another reply:
1. Clients are often CPU bound in the CLIO layers, which I think
would be
unaffected by a larger RPC size.
2. Clients do not package IOs anywhere near as well, resulting in a
larger
number of smaller IOs.
(Even in a test where we do only 1 MB IO requests, on 2.x, we would
see a
large number of
RPCs of (much) < 1 MB from the client, where as in 1.8, we saw
almost
exclusively 1 MB RPCs.
For some tests, we'd see as many as 10 times as many total RPCs
on 2.x vs
1.8.) Since the client isn't doing a good job of filling 1 MB RPCs, I
don't
think it would fill 4 MB RPCs.
In contrast, 2.6 is much more like 1.8. CPU usage is down, and IOs
are
packaged much better. Our IO statistics for 2.6 look much more like 1.8
than for earlier 2.x.
I am using Lustre server 2.6.53 and Lustre client 2.5.3.
Whether Lustre client 2.5.3 has above mentioned issues? Should I use lustre
client 2.6.53 only to see good perf. Results.
--
Mandar Joshi
From: Shuichi Ihara <sihara(a)ddn.com>
To: "Simmons, James A." <simmonsja(a)ornl.gov>
Cc: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
Subject: Re: [HPDD-discuss] Anyone using 4MB RPCs
Message-ID: <E1F361C1-8912-4D12-B0F7-F23656605F27(a)ddn.com>
Content-Type: text/plain; charset="us-ascii"
Yes, we got significant performance improvements with 4MB RPCs. Not only
peak performance, but also high sustained performance even with a lot of
concurrent access to OSTs. max_dirty_mb is one of impotent parameter for 4MB
RPCs, but it's now automatically set to suitable value from
max_pages_per_rpc and max_rpcs_in_flight. (see LU-4933) However, the more
importantly, you should have end to end 4MB IO from client to disk. I mean
clients send to server with 4MB RPCs, but OSS also needs to pass efficient
IO size to OSTs. I believe you are missing this part.
Thanks
Ihara
On Oct 23, 2014, at 10:16 AM, Simmons, James A.
<simmonsja(a)ornl.gov>
wrote:
So recently we have moved our systems from 1.8 to 2.5 clients and have
lost of the performance we had from before which is expected. So I
thought we could try using 4MB RPCs instead of the default 1MB RPC
packet. I set max_pages_per_rpc to 1024 and looked at the value of
max_dirty_mb which was 32 and max_rpcs_in_flight which is 8. By
default a dirty cache of 32MB should be enough in this case. So It
tested it and saw no performance improvements. After that I boosted
max_dirty_mb to 64 and still no improvements over the default settings.
Has anyone
seen this before? What could I be missing?
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
2423
days inactive
2423
days old
0 comments
1 participants
participants (1)
-
Mandar Joshi