Re: [HPDD-discuss] [Lustre-discuss] will obdfilter-survey destroy an already formatted file system
by Dilger, Andreas
On 2013/21/03 4:09 AM, "Michael Kluge" <Michael.Kluge(a)tu-dresden.de> wrote:
>I have read through the documentation for obdfilter-survey but could not
>found any information on how invasive the test is. Will it destroy an
>already formatted OST or render user data unusable?
It shouldn't - the obdfilter-survey uses a different object sequence (2)
compared to normal filesystem objects (currently always 0), so the two do
not collide.
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
8 years, 2 months
2.5 Feature List Request
by Levi, Jodi
Hi all!
As the Lustre 2.4 release draws close to completion, we are turning our thoughts to the content of the Lustre 2.5 release.
Lustre 2.5 is targeted for release in October 2013; therefore, any features to be included will need to be ready for landing to Master in July 2013 in order to allow adequate time for stabilization ahead of the release.
We will be discussing the Support Marix for 2.5 in upcoming CDWG meetings.
If anyone is working on features that could be candidates to include in this release, please let me know so that we can consider them in our planning.
Thank you!
Jodi Levi
Program Manager
Intel High Performance Data Division
mobile: 303-241-1428
email: jodi.levi(a)intel.com<mailto:jodi.levi@intel.com>
8 years, 3 months
Re: [HPDD-discuss] Regarding Lustre Setup for data analytics..
by linux freaker
Wow !!! I am able to run hadoop over lustre.
I started tasktracker at datanode.
This is the complete output now:
# bin/hadoop jar hadoop-examples-1.1.1.jar wordcount
/mnt/lustre/ebook /mnt/lustre/ebook-result12
13/03/21 06:49:00 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/03/21 06:49:00 INFO input.FileInputFormat: Total input paths to process : 17
13/03/21 06:49:00 WARN snappy.LoadSnappy: Snappy native library not loaded
13/03/21 06:49:00 INFO mapred.JobClient: Running job: job_201303191951_0004
13/03/21 06:49:01 INFO mapred.JobClient: map 0% reduce 0%
13/03/21 06:49:41 INFO mapred.JobClient: map 5% reduce 0%
13/03/21 06:49:43 INFO mapred.JobClient: map 11% reduce 0%
13/03/21 06:49:46 INFO mapred.JobClient: map 17% reduce 0%
13/03/21 06:49:48 INFO mapred.JobClient: map 23% reduce 0%
13/03/21 06:49:50 INFO mapred.JobClient: map 23% reduce 7%
13/03/21 06:49:51 INFO mapred.JobClient: map 29% reduce 7%
13/03/21 06:49:53 INFO mapred.JobClient: map 35% reduce 7%
13/03/21 06:49:56 INFO mapred.JobClient: map 41% reduce 7%
13/03/21 06:49:58 INFO mapred.JobClient: map 47% reduce 7%
13/03/21 06:49:59 INFO mapred.JobClient: map 47% reduce 11%
13/03/21 06:50:00 INFO mapred.JobClient: map 52% reduce 11%
13/03/21 06:50:02 INFO mapred.JobClient: map 58% reduce 15%
13/03/21 06:50:04 INFO mapred.JobClient: map 64% reduce 15%
13/03/21 06:50:05 INFO mapred.JobClient: map 64% reduce 19%
13/03/21 06:50:06 INFO mapred.JobClient: map 70% reduce 19%
13/03/21 06:50:08 INFO mapred.JobClient: map 76% reduce 19%
13/03/21 06:50:10 INFO mapred.JobClient: map 82% reduce 19%
13/03/21 06:50:12 INFO mapred.JobClient: map 88% reduce 19%
13/03/21 06:50:14 INFO mapred.JobClient: map 94% reduce 19%
13/03/21 06:50:15 INFO mapred.JobClient: map 94% reduce 29%
13/03/21 06:50:16 INFO mapred.JobClient: map 100% reduce 29%
13/03/21 06:50:23 INFO mapred.JobClient: map 100% reduce 100%
13/03/21 06:50:25 INFO mapred.JobClient: Job complete: job_201303191951_0004
13/03/21 06:50:25 INFO mapred.JobClient: Counters: 27
13/03/21 06:50:25 INFO mapred.JobClient: Job Counters
13/03/21 06:50:25 INFO mapred.JobClient: Launched reduce tasks=1
13/03/21 06:50:25 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=79405
13/03/21 06:50:25 INFO mapred.JobClient: Total time spent by all
reduces wai
ting after reserving slots (ms)=0
13/03/21 06:50:25 INFO mapred.JobClient: Total time spent by all
maps waitin
g after reserving slots (ms)=0
13/03/21 06:50:25 INFO mapred.JobClient: Rack-local map tasks=17
13/03/21 06:50:25 INFO mapred.JobClient: Launched map tasks=17
13/03/21 06:50:25 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=41959
13/03/21 06:50:25 INFO mapred.JobClient: File Output Format Counters
13/03/21 06:50:25 INFO mapred.JobClient: Bytes Written=511158
13/03/21 06:50:25 INFO mapred.JobClient: FileSystemCounters
13/03/21 06:50:25 INFO mapred.JobClient: FILE_BYTES_READ=25105721
13/03/21 06:50:25 INFO mapred.JobClient: FILE_BYTES_WRITTEN=13929334
13/03/21 06:50:25 INFO mapred.JobClient: File Input Format Counters
13/03/21 06:50:25 INFO mapred.JobClient: Bytes Read=18546894
13/03/21 06:50:25 INFO mapred.JobClient: Map-Reduce Framework
13/03/21 06:50:25 INFO mapred.JobClient: Map output materialized
bytes=65396
96
13/03/21 06:50:25 INFO mapred.JobClient: Map input records=410391
13/03/21 06:50:25 INFO mapred.JobClient: Reduce shuffle bytes=6539696
13/03/21 06:50:25 INFO mapred.JobClient: Spilled Records=912072
13/03/21 06:50:25 INFO mapred.JobClient: Map output bytes=31477059
13/03/21 06:50:25 INFO mapred.JobClient: CPU time spent (ms)=70850
13/03/21 06:50:25 INFO mapred.JobClient: Total committed heap
usage (bytes)=
2643394560
13/03/21 06:50:25 INFO mapred.JobClient: Combine input records=3281716
13/03/21 06:50:25 INFO mapred.JobClient: SPLIT_RAW_BYTES=1612
13/03/21 06:50:25 INFO mapred.JobClient: Reduce input records=456036
13/03/21 06:50:25 INFO mapred.JobClient: Reduce input groups=44464
13/03/21 06:50:25 INFO mapred.JobClient: Combine output records=456036
13/03/21 06:50:25 INFO mapred.JobClient: Physical memory (bytes)
snapshot=34
75795968
13/03/21 06:50:25 INFO mapred.JobClient: Reduce output records=44464
13/03/21 06:50:25 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=154
14870016
13/03/21 06:50:25 INFO mapred.JobClient: Map output records=3281716
I wonder if the output really says its running on Lustre and not
HDFS.(just being specific).
Also, I am trying to see how can I reduce the time of complete
execution? Will I need to do anything with stripping of data?
On Sat, Mar 23, 2013 at 10:44 PM, linux freaker <linuxfreaker(a)gmail.com> wrote:
> You said that I just need to start mapred service at master. So, While
> I run jps command I get te following output:
>
> [root@lustreclient1 hadoop]# jps
> 18295 Jps
> 15561 JobTracker
>
> What about the lustreclient2(datanode)? Do I need to run :
> bin/start-all.sh
>
> As of now while I run jps it shows no output which means nothing is
> running at datanode end.
>
> Please suggest.
>
> On 3/20/13, linux freaker <linuxfreaker(a)gmail.com> wrote:
>> I am going to test it today.
>> Thanks for confirming.
>>
>> On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
>>> It would be /mnt/lustre/<some data dir>
>>>
>>> On 3/19/13 9:48 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>>>
>>>>Thanks.
>>>>One more query- I can see that when we run wordcount example in case
>>>>of HDFS, we simply used to run command as:
>>>>
>>>>
>>>>$bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/ebooks
>>>>/user/hduser/ebooks-output
>>>>
>>>>Before which we used to copy Local data into HDFS as shown:
>>>>
>>>>bin/hadoop dfs -copyFromLocal /tmp/ebooks /user/hduser/ebooks.
>>>>
>>>>If I want to run wordcount example in case of lustre, what would be
>>>>the right approach?
>>>>
>>>>Please suggest.
>>>>
>>>>On Tue, Mar 19, 2013 at 9:40 PM, Diep, Minh <minh.diep(a)intel.com> wrote:
>>>>> Yes, that is correct.
>>>>>
>>>>> On 3/19/13 9:05 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>>>>>
>>>>>>Hi,
>>>>>>
>>>>>>Thanks for the quick response.
>>>>>>All I understand is:
>>>>>>
>>>>>>
>>>>>>Master Node (NameNode)
>>>>>>=====================
>>>>>>
>>>>>>File: conf/core-site.xml
>>>>>>
>>>>>><property>
>>>>>><name>fs.default.name</name>
>>>>>><value>file:///</value>
>>>>>></property>
>>>>>><property>
>>>>>>
>>>>>><name>fs.file.impl</name>
>>>>>><value>org.apache.hadoop.fs.LocalFileSystem</value>
>>>>>>
>>>>>><name>hadoop.tmp.dir</name>
>>>>>><value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
>>>>>>
>>>>>>
>>>>>>
>>>>>>File: mapred-site.xml
>>>>>>
>>>>>>
>>>>>><name>mapred.job.tracker</name>
>>>>>><value>lustreclient1:9101</value>
>>>>>>
>>>>>>
>>>>>>
>>>>>>Slave Nodes(DataNodes)
>>>>>>======================
>>>>>>
>>>>>>File: conf/core-site.xml
>>>>>>
>>>>>><property>
>>>>>><name>fs.default.name</name>
>>>>>><value>file:///</value>
>>>>>></property>
>>>>>><property>
>>>>>>
>>>>>><name>fs.file.impl</name>
>>>>>><value>org.apache.hadoop.fs.LocalFileSystem</value>
>>>>>>
>>>>>><name>hadoop.tmp.dir</name>
>>>>>><value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
>>>>>>
>>>>>>File:mapred-site.xml
>>>>>>
>>>>>>In mapred-site.xml:
>>>>>><name>mapred.job.tracker</name>
>>>>>><value>lustreclient1:9101</value> <== Is it correct?
>>>>>>
>>>>>>Please confirm if the entry is correct?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>On Tue, Mar 19, 2013 at 8:30 PM, Diep, Minh <minh.diep(a)intel.com>
>>>>>> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I would suggest you set this instead.
>>>>>>>
>>>>>>>
>>>>>>> <name>fs.default.name</name>
>>>>>>> <value>file:///</value>
>>>>>>>
>>>>>>>
>>>>>>> <name>fs.file.impl</name>
>>>>>>> <value>org.apache.hadoop.fs.LocalFileSystem</value>
>>>>>>>
>>>>>>>
>>>>>>> We set different paths to hadoop.tmp.dir on every node since they are
>>>>>>> sharing the same space.
>>>>>>> On master
>>>>>>> <name>hadoop.tmp.dir</name>
>>>>>>> <value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On slave
>>>>>>> <value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
>>>>>>>
>>>>>>>
>>>>>>> In mapred-site.xml:
>>>>>>> <name>mapred.job.tracker</name>
>>>>>>> <value>client1:9101</value>
>>>>>>>
>>>>>>>
>>>>>>> On master, don't start hdfs since you are using lustre. Start mapred
>>>>>>>only.
>>>>>>>
>>>>>>>
>>>>>>> HTH
>>>>>>> -Minh
>>>>>>>
>>>>>>> On 3/19/13 4:48 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>>>>>>>
>>>>>>>>Hello,
>>>>>>>>
>>>>>>>>I am in verse to setup Hadoop over lustre(replacing HDFS).
>>>>>>>>I have 1 MDS, 2 OSS/OST and 2 Lustre Client.
>>>>>>>>My MDS shows:
>>>>>>>>
>>>>>>>>[code]
>>>>>>>>[root@MDS ~]# lctl list_nids
>>>>>>>>10.84.214.185@tcp
>>>>>>>>[/code]
>>>>>>>>
>>>>>>>>
>>>>>>>>Lustre Client shows:
>>>>>>>>[code]
>>>>>>>>[root@lustreclient1 ~]# lfs df -h
>>>>>>>>UUID bytes Used Available Use% Mounted
>>>>>>>>on
>>>>>>>>lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6%
>>>>>>>>/mnt/lustre[MDT:0]
>>>>>>>>lustre-OST0000_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:0]
>>>>>>>>lustre-OST0001_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:1]
>>>>>>>>lustre-OST0002_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:2]
>>>>>>>>lustre-OST0003_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:3]
>>>>>>>>lustre-OST0004_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:4]
>>>>>>>>lustre-OST0005_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:5]
>>>>>>>>lustre-OST0006_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:6]
>>>>>>>>lustre-OST0007_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:7]
>>>>>>>>lustre-OST0008_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:8]
>>>>>>>>lustre-OST0009_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:9]
>>>>>>>>lustre-OST000a_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:10]
>>>>>>>>lustre-OST000b_UUID 5.9G 276.1M 5.3G 5%
>>>>>>>>/mnt/lustre[OST:11]
>>>>>>>>
>>>>>>>>filesystem summary: 70.9G 3.2G 64.0G 5%
>>>>>>>>/mnt/lustre
>>>>>>>>[/code]
>>>>>>>>
>>>>>>>>Now I installed Hadoop on two Lustre Client(untouching MDS and OSS).
>>>>>>>>
>>>>>>>>My core-site.xml shows:
>>>>>>>>
>>>>>>>>[code]
>>>>>>>><property>
>>>>>>>><name>fs.default.name</name>
>>>>>>>><value>file:///mnt/lustre</value>
>>>>>>>></property>
>>>>>>>><property>
>>>>>>>><name>mapred.system.dir</name>
>>>>>>>><value>${fs.default.name}/hadoop_tmp/mapred/system</value>
>>>>>>>><description>The shared directory where MapReduce stores control
>>>>>>>>files.
>>>>>>>></description>
>>>>>>>></property>
>>>>>>>>[/code]
>>>>>>>>
>>>>>>>>My conf/masters shows
>>>>>>>>[code]
>>>>>>>>lustreclient1
>>>>>>>>[/code]
>>>>>>>>
>>>>>>>>My conf/slaves shows:
>>>>>>>>
>>>>>>>>[code]
>>>>>>>>lustreclient1
>>>>>>>>lustreclient2
>>>>>>>>
>>>>>>>>I have no idea if I need any further configuration file changes.
>>>>>>>>
>>>>>>>>Do I need just the above configuration.
>>>>>>>>What about hdfs-site.xml and mapred-site.xml?
>>>>>>>>_______________________________________________
>>>>>>>>HPDD-discuss mailing list
>>>>>>>>HPDD-discuss(a)lists.01.org
>>>>>>>>https://lists.01.org/mailman/listinfo/hpdd-discuss
>>>>>>>
>>>>>
>>>
>>>
>>
8 years, 4 months
OST000x : Inactive devices error
by linux freaker
Hello,
I had a Lustre Setup working with 1 MDS, 2 OSS/OST and 3 Lustre
Clients. Surprsingly, I checked the filesystem status and it
displayed:
[root@lustreclient1 hadoop]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.5G 278.2M 3.9G 6% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 5.9G 284.1M 5.3G 5% /mnt/lustre[OST:0]
lustre-OST0001_UUID 5.9G 280.4M 5.3G 5% /mnt/lustre[OST:1]
lustre-OST0002_UUID 5.9G 284.9M 5.3G 5% /mnt/lustre[OST:2]
lustre-OST0003_UUID 5.9G 282.8M 5.3G 5% /mnt/lustre[OST:3]
lustre-OST0004_UUID 5.9G 282.8M 5.3G 5% /mnt/lustre[OST:4]
lustre-OST0005_UUID 5.9G 284.1M 5.3G 5% /mnt/lustre[OST:5]
OST0006 : inactive device
OST0007 : inactive device
OST0008 : inactive device
OST0009 : inactive device
OST000a : inactive device
OST000b : inactive device
filesystem summary: 35.4G 1.7G 32.0G 5% /mnt/lustre
I tried to check from other client but it showed correctly:
[root@lustreclient3 ~]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.5G 278.2M 3.9G 6% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 5.9G 284.1M 5.3G 5% /mnt/lustre[OST:0]
lustre-OST0001_UUID 5.9G 280.4M 5.3G 5% /mnt/lustre[OST:1]
lustre-OST0002_UUID 5.9G 284.9M 5.3G 5% /mnt/lustre[OST:2]
lustre-OST0003_UUID 5.9G 282.8M 5.3G 5% /mnt/lustre[OST:3]
lustre-OST0004_UUID 5.9G 282.8M 5.3G 5% /mnt/lustre[OST:4]
lustre-OST0005_UUID 5.9G 284.1M 5.3G 5% /mnt/lustre[OST:5]
lustre-OST0006_UUID 5.9G 281.8M 5.3G 5% /mnt/lustre[OST:6]
lustre-OST0007_UUID 5.9G 282.4M 5.3G 5% /mnt/lustre[OST:7]
lustre-OST0008_UUID 5.9G 282.0M 5.3G 5% /mnt/lustre[OST:8]
lustre-OST0009_UUID 5.9G 283.0M 5.3G 5% /mnt/lustre[OST:9]
lustre-OST000a_UUID 5.9G 284.3M 5.3G 5%
/mnt/lustre[OST:10]
lustre-OST000b_UUID 5.9G 282.5M 5.3G 5%
/mnt/lustre[OST:11]
filesystem summary: 70.9G 3.3G 64.0G 5% /mnt/lustre
I tried unmounting and mounting the /mnt/lustre, disabled firewall,
SELinux disabled, modprobe lustre and modprobe lnet.,
I can ping MDS and OSS/OST . I tried checking at 3rd client too and
everything is working except LustreClient1.
How shall I troubleshoot?
8 years, 4 months
Fwd: Comparing lustre and hadoop distributed filesystem..
by linux freaker
I have a query regarding the File Stripping.
I installed 1 MDS, 2 OSS/OST and 2 LustreClients(would be making it
namenode and datanodes). Now as I run the lfs utility for file
stripping:
[root@lustreclient1 ~]# lfs getstripe /mnt/lustre
/mnt/lustre
stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
/mnt/lustre/ebook
stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
/mnt/lustre/hadoop_tmp
stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
I understand that as of now stripe_count=1 denotes that its stripping
over just 1 OSS/OST.
Since I have 7 OSTs do I need to setstripe count to 7. What is the
exact command, if needed.
As I am comparing Hadoop over Lustre, will striping play a role here?
Please suggest.
On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
>
>
> On 3/20/13 10:54 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>
>>Yes, you are correct. I wrote lustreclient3 twice, instead of
>>lustreclient4.
>>I got your point on this.
>>
>>>while MDS and OSS/OST will be undisturbed and will
>>>neither be namenode/datanode.. am I right?
>>It can be namenode but should not be datanode since we are not
>>recommending mount lustre client on servers.
>>
>>You said it can be namenode. How is it possible?
>>Say, if I take MDS as namenode. In case of namenode, we usually take
>>mount point as /mnt/lustre. But there is no such mount point here in
>>MDS. Same for OSS / OST.
> Namenode only uses a small storage. You could do that with local disk,
> don't need lustre. However, if you use a lustre client as both name node
> and datanode, it's fine too.
>>
>>One more doubt is:
>>
>>You expressed .."If you have 2 oss with 6 OST each, resulting total of
>>12 disks, then you
>>might use 3 disks on each of 4 datanode (ie. Total 12 disks.)
>>However, you are using LVM. That's different."
>>
>>I dint understand why are we concerned regarding 3 disk on each of 4
>>namenode? Are you talking about Hadoop + HDFS here.
> Yes, to try to use comparable resources.
>
>>
>>Though I will go ahead and test the environment, and then come back
>>with more results meanwhile.
>>
>>Thanks for all the suggestion. Its really great to see such an active
>>mailinglist.
> No problem
>
> Thanks
> -Minh
>>
>>
>>
>>
>>On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
>>>
>>>
>>> On 3/20/13 10:21 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>>>
>>>>Just to understand it correctly.
>>>>If I have 1 MDS, 2 OSS with 6 OST each(created through LVM) and 4
>>>>lustreclients.
>>>>So, as per your statement, its equivalent to 1 NameNode(=>
>>>>LustreClient1) and 3 DataNode(=>lustreclient2, lustreclient3,
>>>>lustreclient3),
>>> Should be 4 datanode + 1 namenode. You had lustreclient3 twice?
>>>
>>>>while MDS and OSS/OST will be undisturbed and will
>>>>neither be namenode/datanode.. am I right?
>>> It can be namenode but should not be datanode since we are not
>>> recommending mount lustre client on servers
>>>>
>>>>All I dint get this point .."I would also keep the same total number
>>>>of OSTs and total number of disks on all datanodes." Can you please
>>>>clarify.
>>> If you have 2 oss with 6 OST each, resulting total of 12 disks, then you
>>> might use 3 disks on each of 4 datanode (ie. Total 12 disks.)
>>> However, you are using LVM. That's different.
>>>
>>> You can start out with what you have to see how the perf numbers turn
>>>out,
>>> but it's difficult to draw any conclusion if we are not comparing
>>> apple-to-apple.
>>>
>>> Thanks
>>> -Minh
>>>
>>>>
>>>>Regarding LUG, I will try to see if I can attend it.Thanks for sharing
>>>>it.
>>>>
>>>>On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
>>>>> Hi,
>>>>>
>>>>> There isn't a simple or trivial comparison between Hadoop+HDFS and
>>>>> Hadoop+Lustre.
>>>>> A typical approach (IMHO) is keeping the same number of Lustre client
>>>>>with
>>>>> Hadoop datanode.
>>>>> I would also keep the same total number of OSTs and total number of
>>>>>disks
>>>>> on all datanodes.
>>>>>
>>>>>
8 years, 4 months
Fwd: Regarding Lustre Setup for data analytics..
by linux freaker
You said that I just need to start mapred service at master. So, While
I run jps command I get te following output:
[root@lustreclient1 hadoop]# jps
18295 Jps
15561 JobTracker
What about the lustreclient2(datanode)? Do I need to run :
bin/start-all.sh
As of now while I run jps it shows no output which means nothing is
running at datanode end.
Please suggest.
On 3/20/13, linux freaker <linuxfreaker(a)gmail.com> wrote:
> I am going to test it today.
> Thanks for confirming.
>
> On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
>> It would be /mnt/lustre/<some data dir>
>>
>> On 3/19/13 9:48 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>>
>>>Thanks.
>>>One more query- I can see that when we run wordcount example in case
>>>of HDFS, we simply used to run command as:
>>>
>>>
>>>$bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/ebooks
>>>/user/hduser/ebooks-output
>>>
>>>Before which we used to copy Local data into HDFS as shown:
>>>
>>>bin/hadoop dfs -copyFromLocal /tmp/ebooks /user/hduser/ebooks.
>>>
>>>If I want to run wordcount example in case of lustre, what would be
>>>the right approach?
>>>
>>>Please suggest.
>>>
>>>On Tue, Mar 19, 2013 at 9:40 PM, Diep, Minh <minh.diep(a)intel.com> wrote:
>>>> Yes, that is correct.
>>>>
>>>> On 3/19/13 9:05 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>Thanks for the quick response.
>>>>>All I understand is:
>>>>>
>>>>>
>>>>>Master Node (NameNode)
>>>>>=====================
>>>>>
>>>>>File: conf/core-site.xml
>>>>>
>>>>><property>
>>>>><name>fs.default.name</name>
>>>>><value>file:///</value>
>>>>></property>
>>>>><property>
>>>>>
>>>>><name>fs.file.impl</name>
>>>>><value>org.apache.hadoop.fs.LocalFileSystem</value>
>>>>>
>>>>><name>hadoop.tmp.dir</name>
>>>>><value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
>>>>>
>>>>>
>>>>>
>>>>>File: mapred-site.xml
>>>>>
>>>>>
>>>>><name>mapred.job.tracker</name>
>>>>><value>lustreclient1:9101</value>
>>>>>
>>>>>
>>>>>
>>>>>Slave Nodes(DataNodes)
>>>>>======================
>>>>>
>>>>>File: conf/core-site.xml
>>>>>
>>>>><property>
>>>>><name>fs.default.name</name>
>>>>><value>file:///</value>
>>>>></property>
>>>>><property>
>>>>>
>>>>><name>fs.file.impl</name>
>>>>><value>org.apache.hadoop.fs.LocalFileSystem</value>
>>>>>
>>>>><name>hadoop.tmp.dir</name>
>>>>><value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
>>>>>
>>>>>File:mapred-site.xml
>>>>>
>>>>>In mapred-site.xml:
>>>>><name>mapred.job.tracker</name>
>>>>><value>lustreclient1:9101</value> <== Is it correct?
>>>>>
>>>>>Please confirm if the entry is correct?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>On Tue, Mar 19, 2013 at 8:30 PM, Diep, Minh <minh.diep(a)intel.com>
>>
8 years, 4 months
Fwd: Comparing lustre and hadoop distributed filesystem..
by linux freaker
Yes, you are correct. I wrote lustreclient3 twice, instead of lustreclient4.
I got your point on this.
>while MDS and OSS/OST will be undisturbed and will
>neither be namenode/datanode.. am I right?
It can be namenode but should not be datanode since we are not
recommending mount lustre client on servers.
You said it can be namenode. How is it possible?
Say, if I take MDS as namenode. In case of namenode, we usually take
mount point as /mnt/lustre. But there is no such mount point here in
MDS. Same for OSS / OST.
One more doubt is:
You expressed .."If you have 2 oss with 6 OST each, resulting total of
12 disks, then you
might use 3 disks on each of 4 datanode (ie. Total 12 disks.)
However, you are using LVM. That's different."
I dint understand why are we concerned regarding 3 disk on each of 4
namenode? Are you talking about Hadoop + HDFS here.
Though I will go ahead and test the environment, and then come back
with more results meanwhile.
Thanks for all the suggestion. Its really great to see such an active
mailinglist.
On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
>
>
> On 3/20/13 10:21 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>
>>Just to understand it correctly.
>>If I have 1 MDS, 2 OSS with 6 OST each(created through LVM) and 4
>>lustreclients.
>>So, as per your statement, its equivalent to 1 NameNode(=>
>>LustreClient1) and 3 DataNode(=>lustreclient2, lustreclient3,
>>lustreclient3),
> Should be 4 datanode + 1 namenode. You had lustreclient3 twice?
>
>>while MDS and OSS/OST will be undisturbed and will
>>neither be namenode/datanode.. am I right?
> It can be namenode but should not be datanode since we are not
> recommending mount lustre client on servers
>>
>>All I dint get this point .."I would also keep the same total number
>>of OSTs and total number of disks on all datanodes." Can you please
>>clarify.
> If you have 2 oss with 6 OST each, resulting total of 12 disks, then you
> might use 3 disks on each of 4 datanode (ie. Total 12 disks.)
> However, you are using LVM. That's different.
>
> You can start out with what you have to see how the perf numbers turn out,
> but it's difficult to draw any conclusion if we are not comparing
> apple-to-apple.
>
> Thanks
> -Minh
>
>>
>>Regarding LUG, I will try to see if I can attend it.Thanks for sharing it.
>>
>>On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
>>> Hi,
>>>
>>> There isn't a simple or trivial comparison between Hadoop+HDFS and
>>> Hadoop+Lustre.
>>> A typical approach (IMHO) is keeping the same number of Lustre client
>>>with
>>> Hadoop datanode.
>>> I would also keep the same total number of OSTs and total number of
>>>disks
>>> on all datanodes.
>>>
>>> Please let me know what you find. My finding was that since Lustre is
>>>best
>>> for large file sequential IO, benchmark such as TestDFSIO show Lustre
>>> perform better while terasort does not.
>>>
>>> There will be a talk by Intel at LUG this year about how Intel will
>>> improve Hadoop running on Lustre. Please attend if you have a chance.
>>>
>>> Thanks
>>> -Minh
>>>
>>> On 3/19/13 10:25 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>All I have been reading
>>>>http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdfFile link where
>>>>it talks about how lustre can be more efficient for BigMapOutput kind
>>>>of application.
>>>>I just thought to try my hands comparing HDFS Vs Lustre.
>>>>
>>>>To Test it, I have 1 MDS, 2 OSS/OST and 2 Lustre Client running. My
>>>>Plan is to install Hadoop on 2 Lustre Clients.
>>>>I am going to run wordcount example with this setup
>>>>
>>>>Versus
>>>>
>>>>I have 1 Namenode and 4 DataNode and general Hadoop + HDFS setup.
>>>>
>>>>I wonder how am I going to compare HDFS and Lustre through what number
>>>>of nodes to actually equilibrium the setup?
>>>>
>>>>If I take 1 MDS, 2 OSS/OST and 2 Lustre Client vs 1 Namenode and 4
>>>>DataNode. Will this setup be balanced comparison.
>>>>Please suggest.
>>>>_______________________________________________
>>>>HPDD-discuss mailing list
>>>>HPDD-discuss(a)lists.01.org
>>>>https://lists.01.org/mailman/listinfo/hpdd-discuss
>>>
>>>
>
>
8 years, 4 months
