2:06 a.m.
On 2013-09-29, at 18:49, "Anthony Alba" <ascanio.alba7(a)gmail.com> wrote:
I have the opportunity for a greenfield Lustre 2.5+ site.
We only have on-site experience with 2.1.x and hw RAID ldiskfs.
A major decision point is OSS HBA dumb or RAID, and ldiskfs or ZFS; so could you chime
in on, if you had the choice for 2.5+ would you go for:
1. RAID HBAs or dumb HBAs+RAID enclosure using hw RAID ldiskfs. Would you even consider
hw RAID ZFS?
RAID HBAs are almost certainly not the correct solution, unless you are looking for the
absolute lowest cost solution and have a high tolerance for service interruption. Any
server failure would result in an outage if the disks are only attached to a single server
node.
The dumb HBA+RAID enclosure is by far the most common configuration today. There are a few
common configurations for this in the Lustre world that hopefully others will comment on.
E.g., some sample hardware (as we are mostly a Dell shop)
Dell PERC+MD1200/C8000XD encl,
Dell 6Gbps SAS (dumb) HBA + MD3200 encl
Definitely there are Dell configurations in use at several large sites (e.g. Stampede at
TACC) and I'm sure you can get some info on their configurations. Since there can be
lots of details in getting the configuration right (disks, enclosures, HBAs in specific
PCI slots, nodes, RAM, etc) it makes sense to learn from what others have done, and if
there isn't an overwhelming reason to do differently just copy it.
2. Dumb HBAs with ZFS raidz2
This is a much newer area with less community experience, and there are definitely more
variables here. ZFS is at the "it works reasonably" stage, and we are starting
to pay attention to the performance. Whether you choose ldiskfs or ZFS depends on your
requirements. Currently it is a choice between raw performance vs. data integrity, and
conservative stability vs. longer-term flexibility.
We hope and will work toward ZFS gain ing in performance and deployment experience over
time, but it may still be a bit on the new side for some users.
Cheers, Andreas