On 3/20/13 10:54 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
Yes, you are correct. I wrote lustreclient3 twice, instead of
I got your point on this.
>while MDS and OSS/OST will be undisturbed and will
>neither be namenode/datanode.. am I right?
It can be namenode but should not be datanode since we are not
recommending mount lustre client on servers.
You said it can be namenode. How is it possible?
Say, if I take MDS as namenode. In case of namenode, we usually take
mount point as /mnt/lustre. But there is no such mount point here in
MDS. Same for OSS / OST.
Namenode only uses a small storage. You could do that with
don't need lustre. However, if you use a lustre client as both name node
and datanode, it's fine too.
One more doubt is:
You expressed .."If you have 2 oss with 6 OST each, resulting total of
12 disks, then you
might use 3 disks on each of 4 datanode (ie. Total 12 disks.)
However, you are using LVM. That's different."
I dint understand why are we concerned regarding 3 disk on each of 4
namenode? Are you talking about Hadoop + HDFS here.
Yes, to try to use comparable
Though I will go ahead and test the environment, and then come back
with more results meanwhile.
Thanks for all the suggestion. Its really great to see such an active
On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
> On 3/20/13 10:21 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>>Just to understand it correctly.
>>If I have 1 MDS, 2 OSS with 6 OST each(created through LVM) and 4
>>So, as per your statement, its equivalent to 1 NameNode(=>
>>LustreClient1) and 3 DataNode(=>lustreclient2, lustreclient3,
> Should be 4 datanode + 1 namenode. You had lustreclient3 twice?
>>while MDS and OSS/OST will be undisturbed and will
>>neither be namenode/datanode.. am I right?
> It can be namenode but should not be datanode since we are not
> recommending mount lustre client on servers
>>All I dint get this point .."I would also keep the same total number
>>of OSTs and total number of disks on all datanodes." Can you please
> If you have 2 oss with 6 OST each, resulting total of 12 disks, then you
> might use 3 disks on each of 4 datanode (ie. Total 12 disks.)
> However, you are using LVM. That's different.
> You can start out with what you have to see how the perf numbers turn
> but it's difficult to draw any conclusion if we are not comparing
>>Regarding LUG, I will try to see if I can attend it.Thanks for sharing
>>On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
>>> There isn't a simple or trivial comparison between Hadoop+HDFS and
>>> A typical approach (IMHO) is keeping the same number of Lustre client
>>> Hadoop datanode.
>>> I would also keep the same total number of OSTs and total number of
>>> on all datanodes.
>>> Please let me know what you find. My finding was that since Lustre is
>>> for large file sequential IO, benchmark such as TestDFSIO show Lustre
>>> perform better while terasort does not.
>>> There will be a talk by Intel at LUG this year about how Intel will
>>> improve Hadoop running on Lustre. Please attend if you have a chance.
>>> On 3/19/13 10:25 AM, "linux freaker" <linuxfreaker(a)gmail.com>
>>>>All I have been reading
>>>>http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdfFile link where
>>>>it talks about how lustre can be more efficient for BigMapOutput kind
>>>>I just thought to try my hands comparing HDFS Vs Lustre.
>>>>To Test it, I have 1 MDS, 2 OSS/OST and 2 Lustre Client running. My
>>>>Plan is to install Hadoop on 2 Lustre Clients.
>>>>I am going to run wordcount example with this setup
>>>>I have 1 Namenode and 4 DataNode and general Hadoop + HDFS setup.
>>>>I wonder how am I going to compare HDFS and Lustre through what number
>>>>of nodes to actually equilibrium the setup?
>>>>If I take 1 MDS, 2 OSS/OST and 2 Lustre Client vs 1 Namenode and 4
>>>>DataNode. Will this setup be balanced comparison.
>>>>HPDD-discuss mailing list
HPDD-discuss mailing list