Lustre and kernel buffer interaction
by John Bauer
I have been trying to understand a behavior I am observing in an IOR
benchmark on Lustre. I have pared it down to a simple example.
The IOR benchmark is running in MPI mode. There are 2 ranks, each
running on its own node. Each rank does the following:
Note : Test was run on the "swan" cluster at Cray Inc., using /lus/scratch
write a file. ( 10GB )
fsync the file
close the file
MPI_barrier
open the file that was written by the other rank.
read the file that was written by the other rank.
close the file that was written by the other rank.
The writing of each file goes as expected.
The fsync takes very little time ( about .05 seconds).
The first reads of the file( written by the other rank ) start out *very
*slowly. While theses first reads are proceeding slowly, the
kernel's cached memory ( the Cached: line in /proc/meminfo) decreases
from the size of the file just written to nearly zero.
Once the cached memory has reached nearly zero, the file reading
proceeds as expected.
I have attached a jpg of the instrumentation of the processes that
illustrates this behavior.
My questions are:
Why does the reading of the file, written by the other rank, wait until
the cached data drains to nearly zero before proceeding normally.
Shouldn't the fsync ensure that the file's data is written to the
backing storage so this draining of the cached memory should be simply
releasing pages with no further I/O?
For this case the "dead" time is only about 4 seconds, but this "dead"
time scales directly with the size of the files.
John
--
John Bauer
I/O Doctors LLC
507-766-0378
bauerj(a)iodoctors.com
5 years, 10 months
quotas on 2.4.3
by Matt Bettinger
Hello,
We have a fresh 2.4.3 lustre upgrade that is not yet put into
production running on rhel 6.4.
We would like to take a look at quotas but looks like there is some
major performance problems with 1.8.9 clients.
Here is how I enabled quotas
[root@lfs-mds-0-0 ~]# lctl conf_param lustre2.quota.mdt=ug
[root@lfs-mds-0-0 ~]# lctl conf_param lustre2.quota.ost=ug
[root@lfs-mds-0-0 ~]# lctl get_param osd-*.*.quota_slave.info
osd-ldiskfs.lustre2-MDT0000.quota_slave.info=
target name: lustre2-MDT0000
pool ID: 0
type: md
quota enabled: ug
conn to master: setup
space acct: ug
user uptodate: glb[1],slv[1],reint[0]
group uptodate: glb[1],slv[1],reint[0]
The quotas seem to be working however the write performance from
1.8.9wc client to 2.4.3 with quotas on is horrific. Am I not setting
quotas up correctly?
I try to make a simple user quota on /lustre2/mattb/300MB_QUOTA directory
[root@hous0036 mattb]# lfs setquota -u l0363734 -b 307200 -B 309200 -i
10000 -I 11000 /lustre2/mattb/300MB_QUOTA/
See quota change is in effect...
[root@hous0036 mattb]# lfs quota -u l0363734 /lustre2/mattb/300MB_QUOTA/
Disk quotas for user l0363734 (uid 1378):
Filesystem kbytes quota limit grace files quota limit grace
/lustre2/mattb/300MB_QUOTA/
310292* 307200 309200 - 4 10000 11000 -
Try and write to quota directory as the user but get horrible write speed
[l0363734@hous0036 300MB_QUOTA]$ dd if=/dev/zero of=301MB_FILE bs=1M count=301
301+0 records in
301+0 records out
315621376 bytes (316 MB) copied, 61.7426 seconds, 5.1 MB/s
Try file number 2 and then quota take effect, so it seems.
[l0363734@hous0036 300MB_QUOTA]$ dd if=/dev/zero of=301MB_FILE2 bs=1M count=301
dd: writing `301MB_FILE2': Disk quota exceeded
dd: closing output file `301MB_FILE2': Input/output error
If I disable quotas using
[root@lfs-mds-0-0 ~]# lctl conf_param lustre2.quota.mdt=none
[root@lfs-mds-0-0 ~]# lctl conf_param lustre2.quota.oss=none
Then try and write the same file the speeds are more like we expect
but then can't use quotas.
[l0363734@hous0036 300MB_QUOTA]$ dd if=/dev/zero of=301MB_FILE2 bs=1M count=301
301+0 records in
301+0 records out
315621376 bytes (316 MB) copied, 0.965009 seconds, 327 MB/s
[l0363734@hous0036 300MB_QUOTA]$ dd if=/dev/zero of=301MB_FILE2 bs=1M count=301
I have not tried this with a 2.4 client, yet since all of our nodes
are 1.8.X until we rebuild our images.
I was going by the manual on
http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact...
but it looks like I am running into interoperability issue (which I
thought I fixed by using 1.8.9-wc client) or just not configuring
this correctly.
Thanks!
MB
6 years, 1 month
MDS kernel panic
by Pardo Diaz, Alfonso
Hello,
Since I update my lustre 2.2 to 2.5.1 (Centos6.5) and copy the MDT to a new SSD disk. I get random kernel panics in the MDS (both HA pairs). The last kernel panic I get this log:
<4>Lustre: MGS: non-config logname received: params
<3>LustreError: 11-0: cetafs-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
<4>Lustre: MGS: non-config logname received: params
<4>Lustre: cetafs-MDT0000: Will be in recovery for at least 5:00, or until 102 clients reconnect
<4>Lustre: MGS: non-config logname received: params
<4>Lustre: MGS: non-config logname received: params
<4>Lustre: Skipped 5 previous similar messages
<4>Lustre: MGS: non-config logname received: params
<4>Lustre: Skipped 9 previous similar messages
<4>Lustre: MGS: non-config logname received: params
<4>Lustre: Skipped 2 previous similar messages
<4>Lustre: MGS: non-config logname received: params
<4>Lustre: Skipped 23 previous similar messages
<4>Lustre: MGS: non-config logname received: params
<4>Lustre: Skipped 8 previous similar messages
<3>LustreError: 3461:0:(ldlm_lib.c:1751:check_for_next_transno()) cetafs-MDT0000: waking for gap in transno, VBR is OFF (skip: 17188113481, ql: 1, comp: 101, conn: 102, next: 17188113493, last_committed: 17188113480)
<6>Lustre: cetafs-MDT0000: Recovery over after 1:13, of 102 clients 102 recovered and 0 were evicted.
<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffffa0c3b6a0>] __iam_path_lookup+0x70/0x1f0 [osd_ldiskfs]
<4>PGD 106c0bf067 PUD 106c0be067 PMD 0
<4>Oops: 0002 [#1] SMP
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 0
<4>Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) ipmi_devintf cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 dm_multipath microcode iTCO_wdt iTCO_vendor_support sb_edac edac_core lpc_ich mfd_core i2c_i801 igb i2c_algo_bit i2c_core ptp pps_core ioatdma dca mlx4_ib ib_sa ib_mad ib_core mlx4_en mlx4_core sg ext4 jbd2 mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>
<4>Pid: 3362, comm: mdt00_001 Not tainted 2.6.32-431.5.1.el6_lustre.x86_64 #1 Bull SAS bullx/X9DRH-7TF/7F/iTF/iF
<4>RIP: 0010:[<ffffffffa0c3b6a0>] [<ffffffffa0c3b6a0>] __iam_path_lookup+0x70/0x1f0 [osd_ldiskfs]
<4>RSP: 0018:ffff88085e2754b0 EFLAGS: 00010246
<4>RAX: 00000000fffffffb RBX: ffff88085e275600 RCX: 000000000009c93c
<4>RDX: 0000000000000000 RSI: 000000000009c93b RDI: ffff88106bcc32f0
<4>RBP: ffff88085e275500 R08: 0000000000000000 R09: 00000000ffffffff
<4>R10: 0000000000000000 R11: 0000000000000000 R12: ffff88085e2755c8
<4>R13: 0000000000005250 R14: ffff8810569bf308 R15: 0000000000000001
<4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
<4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 0000000000000000 CR3: 000000106dd9b000 CR4: 00000000000407f0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process mdt00_001 (pid: 3362, threadinfo ffff88085e274000, task ffff88085f55c080)
<4>Stack:
<4> 0000000000000000 ffff88085e2755d8 ffff8810569bf288 ffffffffa00fd2c4
<4><d> ffff88085e275660 ffff88085e2755c8 ffff88085e2756c8 0000000000000000
<4><d> 0000000000000000 ffff88085db2a480 ffff88085e275530 ffffffffa0c3ba6c
<4>Call Trace:
<4> [<ffffffffa00fd2c4>] ? do_get_write_access+0x3b4/0x520 [jbd2]
<4> [<ffffffffa0c3ba6c>] iam_lookup_lock+0x7c/0xb0 [osd_ldiskfs]
<4> [<ffffffffa0c3bad4>] __iam_it_get+0x34/0x160 [osd_ldiskfs]
<4> [<ffffffffa0c3be1e>] iam_it_get+0x2e/0x150 [osd_ldiskfs]
<4> [<ffffffffa0c3bf4e>] iam_it_get_exact+0xe/0x30 [osd_ldiskfs]
<4> [<ffffffffa0c3d47f>] iam_insert+0x4f/0xb0 [osd_ldiskfs]
<4> [<ffffffffa0c366ea>] osd_oi_iam_refresh+0x18a/0x330 [osd_ldiskfs]
<4> [<ffffffffa0c3ea40>] ? iam_lfix_ipd_alloc+0x0/0x20 [osd_ldiskfs]
<4> [<ffffffffa0c386dd>] osd_oi_insert+0x11d/0x480 [osd_ldiskfs]
<4> [<ffffffff811ae522>] ? generic_setxattr+0xa2/0xb0
<4> [<ffffffffa0c25021>] ? osd_ea_fid_set+0xf1/0x410 [osd_ldiskfs]
<4> [<ffffffffa0c33595>] osd_object_ea_create+0x5b5/0x700 [osd_ldiskfs]
<4> [<ffffffffa0e173bf>] lod_object_create+0x13f/0x260 [lod]
<4> [<ffffffffa0e756c0>] mdd_object_create_internal+0xa0/0x1c0 [mdd]
<4> [<ffffffffa0e86428>] mdd_create+0xa38/0x1730 [mdd]
<4> [<ffffffffa0c2af37>] ? osd_xattr_get+0x97/0x2e0 [osd_ldiskfs]
<4> [<ffffffffa0e14770>] ? lod_index_lookup+0x0/0x30 [lod]
<4> [<ffffffffa0d50358>] mdo_create+0x18/0x50 [mdt]
<4> [<ffffffffa0d5a64c>] mdt_reint_open+0x13ac/0x21a0 [mdt]
<4> [<ffffffffa065983c>] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc]
<4> [<ffffffffa04f4600>] ? lu_ucred_key_init+0x160/0x1a0 [obdclass]
<4> [<ffffffffa0d431f1>] mdt_reint_rec+0x41/0xe0 [mdt]
<4> [<ffffffffa0d2add3>] mdt_reint_internal+0x4c3/0x780 [mdt]
<4> [<ffffffffa0d2b35d>] mdt_intent_reint+0x1ed/0x520 [mdt]
<4> [<ffffffffa0d26a0e>] mdt_intent_policy+0x3ae/0x770 [mdt]
<4> [<ffffffffa0610511>] ldlm_lock_enqueue+0x361/0x8c0 [ptlrpc]
<4> [<ffffffffa0639abf>] ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
<4> [<ffffffffa0d26ed6>] mdt_enqueue+0x46/0xe0 [mdt]
<4> [<ffffffffa0d2dbca>] mdt_handle_common+0x52a/0x1470 [mdt]
<4> [<ffffffffa0d68545>] mds_regular_handle+0x15/0x20 [mdt]
<4> [<ffffffffa0669a45>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
<4> [<ffffffffa03824ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
<4> [<ffffffffa03933df>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
<4> [<ffffffffa06610e9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
<4> [<ffffffff81054839>] ? __wake_up_common+0x59/0x90
<4> [<ffffffffa066adad>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
<4> [<ffffffffa066a2c0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
<4> [<ffffffff8109aee6>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff8109ae50>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20
<4>Code: 00 48 8b 5d b8 45 31 ff 0f 1f 00 49 8b 46 30 31 d2 48 89 d9 44 89 ee 48 8b 7d c0 ff 50 20 48 8b 13 66 2e 0f 1f 84 00 00 00 00 00 <f0> 0f ba 2a 19 19 c9 85 c9 74 15 48 8b 0a f7 c1 00 00 00 02 74
<1>RIP [<ffffffffa0c3b6a0>] __iam_path_lookup+0x70/0x1f0 [osd_ldiskfs]
<4> RSP <ffff88085e2754b0>
<4>CR2: 0000000000000000
Any suggestion is welcome?
THANKS!!!
Alfonso Pardo Diaz
System Administrator / Researcher
c/ Sola nº 1; 10200 Trujillo, ESPAÑA
Tel: +34 927 65 93 17 Fax: +34 927 32 32 37
----------------------------
Confidencialidad:
Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente respondiendo al mensaje y proceda a su destrucción.
Disclaimer:
This message and its attached files is intended exclusively for its recipients and may contain confidential information. If you received this e-mail in error you are hereby notified that any dissemination, copy or disclosure of this communication is strictly prohibited and may be unlawful. In this case, please notify us by a reply and delete this email and its contents immediately.
----------------------------
6 years, 8 months
multiple networks and lnet
by Michael Di Domenico
>From my understanding of the lustre manual, I'm not sure what I want
to do is possible, so I figured i'd ask
I have several clusters on different subnets
clusterA eth0 192.168.1.0/24 ib0 192.168.2.0/24
clusterB eth0 192.168.3.0/24 ib0 192.168.4.0/24
...etc...
what i want to do is connect lustre onto a separate network and use
only infiniband for the lustre communication
lustreA eth0 192.168.5.0/24 ib0 192.168.6.0/24
if i add
ib0:1 192.168.2.250/32
ib0:2 192.168.4.250/32
to the lustreA server I can talk via ipoib. however as i understand
it (probably incorrectly) lustreA would appear in lnet as
o2ib0(ib0) 192.168.6.250
o2ib1(ib0:1) 192.168.2.250
o2ib2(ib0:2) 192.168.4.250
however the clients would appear as
clusterA o2ib0(ib0) 192.168.2.250
clusterB o2ib0(ib0) 192.168.4.250
which as i understand it, creates a conflict in the o2ib devices where
o2ib0 on the client will not match o2ib2(ib0:2) on the lustre server
Is there a way to accomplish this or a better way overall?
6 years, 9 months
Is upgrade from 2.1.6 to 2.5.1 safe?
by Oliver Mangold
Hi,
we consider upgrading a production system from Lustre 2.1.6 to 2.5.1.
According to some recent conversation here, it seems this should be done
via version 2.4.2. Is this correct?
I am worried about the safety of the data. Is this upgrade path now a
well-tested procedure known to work? Or shouldn't this be done with
important data?
Best,
Oliver Mangold
--
Dr. Oliver Mangold
System Analyst
NEC Deutschland GmbH
HPC Division
Hessbrühlstraße 21b
70565 Stuttgart
Germany
Phone: +49 711 78055 13
Mail: oliver.mangold(a)emea.nec.com
6 years, 9 months
MDTs recovering or not recovering
by Javed Shaikh
CentOS 6.4 / Lustre 2.4.2 (both client and servers)
hi,
it looks like MDTs are not recovering after more than 12hours of being in that state.
there's hardly any activity happening on the MDS.
what would happen if the recovery is aborted through lctl?
thanks,
javed
6 years, 9 months
Unusable ZFS backup MDT/MGS after change
by Jesse Stroik
(I apologize to those who are also lustre-discuss members as this is a
cross-post)
We have been using the following method to maintain a backup MGS/MDS:
https://jira.hpdd.intel.com/browse/LUDOC-161
The procedure we follow is commented by Scott and was taken from our own
internal documentation. Our method is to create a ZFS file systems:
lustre-meta/fsmeta
lustre-meta/mgs
And then we snapshot (-r), then zfs send (-R)/receive it. To use the
backup we swap its IP addresses and name with the primary.
If the backup is swapped and used immediately after the snapshot is
taken, this method works. If you continue to use the original server
before migrating to the backup, it does not work. This is the equivalent
of migrating to a not-perfectly-up-to-date snapshot.
In the broken case, we can still read files. We cannot create new files.
The MDS does show files we attempt to create but their attributes are
all unknown. We are unable to manipulate those files (rm, mv, etc)
because they do not exist on the OSTs. The error returned is "cannot
allocate memory" (see LU-4524).
We suspected the configuration logs and so we re-ran writeconf and
remounted. Same behavior.
We are running lustre 2.4.0 on the servers and have tested with 2.4.0
and 2.1.6 clients.
Best,
Jesse Stroik
University of Wisconsin
6 years, 9 months
Upgrading from Lustre 2.1.5 to Lustre 2.5.0
by Roger Spellman
Hi,
I formatted a file system with Lustre 2.1.5, and have used it for a while. I just upgraded to Lustre 2.5.0. I thought that I should be able to mount all the targets. I was able to mount the MDT without a problem. When I try to mount an OST, I get this error:
LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. quota=off. Opts:
Lustre: srv-hss55-OST0002: No data found on store. Initialize space
LustreError: 48809:0:(obd_config.c:1341:class_process_proc_param()) hss55-OST0002: unknown param read_cache_enable=0
LustreError: 48809:0:(obd_config.c:1591:class_config_llog_handler()) MGC10.2.99.21@o2ib: cfg command failed: rc = -38
Lustre: cmd=cf00f 0:hss55-OST0002 1:ost.read_cache_enable=0
LustreError: 15c-8: MGC10.2.99.21@o2ib: The configuration from log 'hss55-OST0002' failed (-38). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
LustreError: 48614:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server hss55-OST0002: -38
LustreError: 48614:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -38
LustreError: 48614:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) hss55-MDT0000-lwp-OST0002: Can't end config log hss55-client.
LustreError: 48614:0:(obd_mount_server.c:1426:server_put_super()) hss55-OST0002: failed to disconnect lwp. (rc=-2)
Lustre: Failing over hss55-OST0002
Lustre: server umount hss55-OST0002 complete
LustreError: 48614:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount (-38)
mount.lustre: mount /dev/mapper/map02 at /mnt/ost2 failed: Function not implemented
Should this have worked? Any advice on how to fix this?
Thanks.
Roger Spellman
Sr. Staff Engineer
Terascala, Inc.
508-588-1501
www.terascala.com http://www.terascala.com/
6 years, 9 months
FW: Facing issues in applying zero copy patch on rhel6.
by Mandar Joshi
Hi,
I and Aayush were investigating more on point 3 in previous mail. What we
understood that obdfilter was replaced with ofd module in Lustre 2.4.0.
I found a patch which removes zero copy changes from lustre:
<https://github.com/rread/lustre/commit/0c69ea9f6d6d044b6463787d5a56b2384bb9
663a#diff-622f0cb8b18627484c80ad225d1803de>
https://github.com/rread/lustre/commit/0c69ea9f6d6d044b6463787d5a56b2384bb96
63a#diff-622f0cb8b18627484c80ad225d1803de
As asked before in #1 in previous mail, we could not find on any mailing
list any specific reason/known problem for removing this support.
We are working on porting RAID driver zcp changes to kernel wrt Lustre
2.5.0. We are also planning to add and test these removed changes back in
lustre 2.5.0 as part of zero copy changes porting in lustre.
Will it be right to do so? Any suggestion or precaution which we should take
while porting these changes?
Thanks,
Mandar & Aayush.
PS: Adding this mail thread on hpdd-discuss as well.
-------- Original Message --------
Subject:
Re: Facing issues in applying zero copy patch on rhel6.
Date:
Mon, 07 Apr 2014 17:46:03 +0530
From:
aayush agrawal <mailto:aayush.agrawal@calsoftinc.com>
<aayush.agrawal(a)calsoftinc.com>
To:
lustre-discuss(a)lists.lustre.org
Hi,
Following are few questions which still remain unanswered:
1. I am seeing that zero copy patch existed in lustre-2.1.2, In
lustre/kernel-patches/patches directory for rhel5. But it has been removed
in latest lustre 2.5.0. Is there any specific reason to remove it?
2. If I am correct this code gets invoked from following location from
obdfilter(correct me if I am wrong):
1. lustre/obdfilter/filter_io_26.c line no:367
;& nbsp;
&n bsp;&nb sp; <snip>
/* I only set the page to be constant only if it
* is mapped to a contiguous underlying disk
block(s).
* It will then make sure the corresponding device
* cache of raid5 will be overwritten by this page.
* - jay */
if ((rw == OBD_BRW_WRITE) &&
(nblocks == blocks_per_page) &&
mapping_cap_page_constant_write(inode->i_mapping))
SetPageConstant(page);
</snip>
3. I want to use this zero copy patch in latest lustre 2.5.0(With
kernel linux-2.6.32-358.18.1.el6). I could port patch to this latest kernel.
But it looks like obdfilter module has been replaced in latest kernel which
used to invoke zero copy changes as mentioned in point 2 I think
its ofd module with which it got replaced(??). So latest lustre is not able
to invoke zero copy changes(for eg: setting/clearing PG_constant etc.). Is
there any different patch available for invoking these zero copy changes?
4. Can you also suggest which newly added module in lustre-2.5.0
would require these changes?
Thanks,
Aayush
On 4/7/2014 4:50 PM, aayush agrawal wrote:
Hi,
I got both the errors resolved. Looks like there was some issue with my
lustre build process itself.
So I think along with raid5-zerocopy-rhel6.patch, raid5-stats-rhel6.patch
was the only additional patch required.
Thanks,
Aayush
On 3/24/2014 8:40 PM, aayush agrawal wrote:
Hi,
I wanted to apply zero copy patch on rhel6 so I followed below steps:
1. Downloaded lustre-2.1.2 and corresponding kernel
rpm(linux-2.6.32-220.17.1.el6). The OS I am using is CentOS 6.4.
2. Applied kernel patches from lustre source code to kernel.
3. Downloaded a zero copy patch for rhel6 from:
https://github.com/Xyratex/lustre-stable/blob/b_neo_1.4.0/lustre/kernel_patc
hes/patches/raid5-zerocopy-rhel6.patch
4. Applied this patch to above kernel.
5. Then I tried to compile this kernel but it gives undefined symbol
errors for writes_zcopy and PageConstant.
1. For writes_zcopy I found another patch on the same git hub link:
https://github.com/Xyratex/lustre-stable/blob/b_neo_1.4.0/lustre/kernel_patc
hes/patches/raid5-stats-rhel6.patch.
2. For PageConstant I couldn't find any patch to rectify this error.
Even I do not see usage of PG_Constant/SetPageConstant etc. in
raid5-zerocopy-rhel6.patch which I think is essential.
6. So my questions are:
1. I think few patches are missing here. As mentioned above one of them
would be raid5-stats-rhel6.patch please confirm.
2. if confirmed, still there has to be at least one patch missing (to
include PageConstant use). Are there any other patches to be applied before
I apply raid5-stats-rhel6.patch and raid5-zerocopy-rhel6.patch.
3. I am seeing that this zero copy patch existed in lustre-2.1.2, In
lustre/kernel-patches/patches directory for rhel5. But it has been removed
in latest lustre 2.5.0. Is there any specific reason to remove it.
Thanks,
Aayush
6 years, 9 months
Remove filesystem directories from MDT
by Pardo Diaz, Alfonso
Hello,
I have some problems in my filesystem. When I browse the filesystem from a client, a specific directory have directories that contain the same directories, in other words:
LustreFS-> dir A -> dir B -> dir B -> dir B -> dir B -> dir B…
This directory, and its children, has the same obdidx/objid:
[root@client vm-106-disk-1.raw]# lfs getstripe vm-106-disk-1.raw
vm-106-disk-1.raw/vm-106-disk-1.raw
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: 1
lmm_layout_gen: 0
lmm_stripe_offset: 19
obdidx objid objid group
19 7329413 0x6fd685 0
[root@client vm-106-disk-1.raw]# cd vm-106-disk-1.raw/
[root@client vm-106-disk-1.raw]# pwd
/mnt/data/106/GNUSparseFile.2227/vm-106-disk-1.raw/vm-106-disk-1.raw/vm-106-disk-1.raw
[root@client vm-106-disk-1.raw]# lfs getstripe vm-106-disk-1.raw
vm-106-disk-1.raw/vm-106-disk-1.raw
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: 1
lmm_layout_gen: 0
lmm_stripe_offset: 19
obdidx objid objid group
19 7329413 0x6fd685 0
[root@client vm-106-disk-1.raw]# cd vm-106-disk-1.raw/
[root@client vm-106-disk-1.raw]# pwd
/mnt/data/106/GNUSparseFile.2227/vm-106-disk-1.raw/vm-106-disk-1.raw/vm-106-disk-1.raw/vm-106-disk-1.raw
[root@client vm-106-disk-1.raw]# lfs getstripe vm-106-disk-1.raw
vm-106-disk-1.raw/vm-106-disk-1.raw
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: 1
lmm_layout_gen: 0
lmm_stripe_offset: 19
obdidx objid objid group
19 7329413 0x6fd685 0
But when I mount the MDT with “Ldiskfs” I can see correctly the filesystem in the “ROOT” mdt directory.
I wish to remove from the client this looped directory (dir B in the example), but when I try to remove I get a “kernel panic" in the MDS/MGS.
is it a good idea remove a subdirectory in the MDT (with Ldiskfs mounted) in the “ROOT” mdt directory or I will get orphan objects in the OST, if I remove this directory in the MDT?
Thanks!!!
Alfonso Pardo Diaz
System Administrator / Researcher
c/ Sola nº 1; 10200 Trujillo, ESPAÑA
Tel: +34 927 65 93 17 Fax: +34 927 32 32 37
----------------------------
Confidencialidad:
Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente respondiendo al mensaje y proceda a su destrucción.
Disclaimer:
This message and its attached files is intended exclusively for its recipients and may contain confidential information. If you received this e-mail in error you are hereby notified that any dissemination, copy or disclosure of this communication is strictly prohibited and may be unlawful. In this case, please notify us by a reply and delete this email and its contents immediately.
----------------------------
6 years, 9 months
