Fwd: Comparing lustre and hadoop distributed filesystem..
by linux freaker
Just to understand it correctly.
If I have 1 MDS, 2 OSS with 6 OST each(created through LVM) and 4 lustreclients.
So, as per your statement, its equivalent to 1 NameNode(=>
LustreClient1) and 3 DataNode(=>lustreclient2, lustreclient3,
lustreclient3),while MDS and OSS/OST will be undisturbed and will
neither be namenode/datanode.. am I right?
All I dint get this point .."I would also keep the same total number
of OSTs and total number of disks on all datanodes." Can you please
clarify.
Regarding LUG, I will try to see if I can attend it.Thanks for sharing it.
On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
> Hi,
>
> There isn't a simple or trivial comparison between Hadoop+HDFS and
> Hadoop+Lustre.
> A typical approach (IMHO) is keeping the same number of Lustre client with
> Hadoop datanode.
> I would also keep the same total number of OSTs and total number of disks
> on all datanodes.
>
> Please let me know what you find. My finding was that since Lustre is best
> for large file sequential IO, benchmark such as TestDFSIO show Lustre
> perform better while terasort does not.
>
> There will be a talk by Intel at LUG this year about how Intel will
> improve Hadoop running on Lustre. Please attend if you have a chance.
>
> Thanks
> -Minh
>
> On 3/19/13 10:25 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>
>>Hi,
>>
>>All I have been reading
>>http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdfFile link where
>>it talks about how lustre can be more efficient for BigMapOutput kind
>>of application.
>>I just thought to try my hands comparing HDFS Vs Lustre.
>>
>>To Test it, I have 1 MDS, 2 OSS/OST and 2 Lustre Client running. My
>>Plan is to install Hadoop on 2 Lustre Clients.
>>I am going to run wordcount example with this setup
>>
>>Versus
>>
>>I have 1 Namenode and 4 DataNode and general Hadoop + HDFS setup.
>>
>>I wonder how am I going to compare HDFS and Lustre through what number
>>of nodes to actually equilibrium the setup?
>>
>>If I take 1 MDS, 2 OSS/OST and 2 Lustre Client vs 1 Namenode and 4
>>DataNode. Will this setup be balanced comparison.
>>Please suggest.
>>_______________________________________________
>>HPDD-discuss mailing list
>>HPDD-discuss(a)lists.01.org
>>https://lists.01.org/mailman/listinfo/hpdd-discuss
>
>
8 years, 8 months
Comparing lustre and hadoop distributed filesystem..
by linux freaker
Hi,
All I have been reading
http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdfFile link where
it talks about how lustre can be more efficient for BigMapOutput kind
of application.
I just thought to try my hands comparing HDFS Vs Lustre.
To Test it, I have 1 MDS, 2 OSS/OST and 2 Lustre Client running. My
Plan is to install Hadoop on 2 Lustre Clients.
I am going to run wordcount example with this setup
Versus
I have 1 Namenode and 4 DataNode and general Hadoop + HDFS setup.
I wonder how am I going to compare HDFS and Lustre through what number
of nodes to actually equilibrium the setup?
If I take 1 MDS, 2 OSS/OST and 2 Lustre Client vs 1 Namenode and 4
DataNode. Will this setup be balanced comparison.
Please suggest.
8 years, 8 months
Regarding Lustre Setup for data analytics..
by linux freaker
Hello,
I am in verse to setup Hadoop over lustre(replacing HDFS).
I have 1 MDS, 2 OSS/OST and 2 Lustre Client.
My MDS shows:
[code]
[root@MDS ~]# lctl list_nids
10.84.214.185@tcp
[/code]
Lustre Client shows:
[code]
[root@lustreclient1 ~]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:0]
lustre-OST0001_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:1]
lustre-OST0002_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:2]
lustre-OST0003_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:3]
lustre-OST0004_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:4]
lustre-OST0005_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:5]
lustre-OST0006_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:6]
lustre-OST0007_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:7]
lustre-OST0008_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:8]
lustre-OST0009_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:9]
lustre-OST000a_UUID 5.9G 276.1M 5.3G 5%
/mnt/lustre[OST:10]
lustre-OST000b_UUID 5.9G 276.1M 5.3G 5%
/mnt/lustre[OST:11]
filesystem summary: 70.9G 3.2G 64.0G 5% /mnt/lustre
[/code]
Now I installed Hadoop on two Lustre Client(untouching MDS and OSS).
My core-site.xml shows:
[code]
<property>
<name>fs.default.name</name>
<value>file:///mnt/lustre</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>${fs.default.name}/hadoop_tmp/mapred/system</value>
<description>The shared directory where MapReduce stores control
files.
</description>
</property>
[/code]
My conf/masters shows
[code]
lustreclient1
[/code]
My conf/slaves shows:
[code]
lustreclient1
lustreclient2
I have no idea if I need any further configuration file changes.
Do I need just the above configuration.
What about hdfs-site.xml and mapred-site.xml?
8 years, 8 months
Not able to mount the Lustre file system
by Linov Suresh
I'm using Scientific Linux release 6.1 (Carbon) Lustre 2.2. I tried to
mount the client I got the error--
[root@nexus net_dugan]# mount -t lustre 10.0.0.1@tcp0:/netfs /lustre
mount.lustre: mount 10.0.0.1@tcp0:/netfs at /lustre failed: No such file or
directory
Is the MGS specification correct?
Is the filesystem name correct?
If upgrading, is the copied client log valid? (see upgrade docs)
This is my device list on my metadata server.
[root@metadata net_linov]# lctl dl
0 UP mgs MGS MGS 17
1 UP mgc MGC10.0.0.1@tcp 45e4df62-b918-6b19-5752-73ca2f6359f5 5
2 UP lov netfs-MDT0000-mdtlov netfs-MDT0000-mdtlov_UUID 4
3 UP mdt netfs-MDT0000 netfs-MDT0000_UUID 5
4 UP mds mdd_obd-netfs-MDT0000 mdd_obd_uuid-netfs-MDT0000 3
5 UP osc netfs-OST000a-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
6 UP osc netfs-OST0001-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
7 UP osc netfs-OST0004-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
* 8 AT osc netfs-OST0005-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 1*
9 UP osc netfs-OST0002-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
10 UP osc netfs-OST0006-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
11 UP osc netfs-OST0007-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
12 UP osc netfs-OST0003-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
13 UP osc netfs-OST000b-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
14 UP osc netfs-OST0009-osc-MDT0000 netfs-MDT0000-mdtlov_UUID 5
*Just wondering why the OST0005 is in AT state?* Because of the AT state
I'm not able to do mount the file system. By mistake I have deleted the
pool information as well. All I want is just retrieve the data.
I ran write-conf on metadata server, ran e2fsprogs as well, still no luck.
Can anyone help me on this?
Thanks,
Linov.
8 years, 8 months
wiki.whamcloud.com/jira.whamcloud.com upgrade/transition right now
by Joshua J. Kugler
(This is a follow-up email to remind all wiki and JIRA users of an impending
upgrade/transition which will result in brief service interruptions for the
next few hours. The text of the original email follows: )
This e-mail is to let you know about upcoming changes to two key
Whamcloud/HPDD services. We will be upgrading the installations of Confluence
(wiki) and Jira (bug tracker), and moving them to Intel's servers. This change
will be happening on March 16, 2013 starting at 3:00AM GMT (March 15, 2013
8:00PM MDT). A follow-up email will be sent out once the changes are complete.
During this change we will have some down time (no more than three hours)
while we transfer the contents to the Intel servers and upgrade the
application software. The upgraded versions of JIRA and Confluence include some
relatively minor user-visible changes, but we do not anticipate that the
upgrade will necessitate any changes in work flow. When the sites come back up,
you will be automatically redirected to the new URLs.
Please contact joshua.kugler(a)intel.com with any questions or concerns.
--
Dev/Ops Lead
High Performance Data Division (formerly Whamcloud)
Intel
8 years, 8 months
wiki.whamcloud.com and jira.whamcloud.com upgrade/transition
by Joshua J. Kugler
This e-mail is to let you know about upcoming changes to two key
Whamcloud/HPDD services. We will be upgrading the installations of Confluence
(wiki) and Jira (bug tracker), and moving them to Intel's servers. This change
will be happening on March 16, 2013 starting at 3:00AM GMT (March 15, 2013
8:00PM MDT). A follow-up email will be sent out once the changes are complete.
During this change we will have some down time (no more than three hours)
while we transfer the contents to the Intel servers and upgrade the
application software. The upgraded versions of JIRA and Confluence include some
relatively minor user-visible changes, but we do not anticipate that the
upgrade will necessitate any changes in work flow. When the sites come back up,
you will be automatically redirected to the new URLs.
Please contact joshua.kugler(a)intel.com with any questions or concerns.
Thanks!
j
--
Dev/Ops Lead
High Performance Data Division (formerly Whamcloud)
Intel
8 years, 8 months
hung lustre file system
by Stu Midgley
Evening
A mistake was made during the addition of 1 oss (2 ost's) to our online
lustre file system.
The ost was formatted with
mkfs.lustre --ost --mgsnode=172.29.0.251 --fsname=l1 --index 14
--mkfsoptions="-T largefile -m 0.5 -E stride=16,stripe-width=160" /dev/sdb
and then mounted... and then quickly unmounted... and then reformatted with
the same command and then attempted to be remounted...
We now have a hung file system.
on the mds I see
Mar 12 13:38:09 mds kernel: LustreError: 140-5: Server l1-OST000e requested
index 14, but that index is already in use. Use --writeconf to force
now what should I do? Bring everything down, writeconf all the servers and
bring everything back up? reformat the ost? lfs df -h on a client is
showing
OST000e : inactive device
Thanks.
--
Dr Stuart Midgley
sdm900(a)sdm900.com
8 years, 8 months
OpenSFS Survey 2013
by Hamilton, Pam
Dear Lustre Community,
The OpenSFS Community Development Working Group has launched a second survey for organizations using Lustre. We are looking for trends in Lustre usage to assist with future planning on releases and will present the results at LUG.
Please complete this short survey (http://www.surveymonkey.com/s/LWQC8LM) to make sure your organization's voice is heard!
Response to the survey is due by February 28th. Note that all questions are optional so it is ok to submit a partially completed survey if you prefer not to disclose some information.
Best regards,
Pam Hamilton
OpenSFS CDWG Lead
___________________________________
Pam Hamilton
Lawrence Livermore National Lab
P.O. Box 808, L-556
Livermore, CA 94551-9900
E-Mail: pgh(a)llnl.gov
Phone: 925-423-1332 Fax: 925-423-8719
8 years, 8 months
