Is it possible: two separate filesystems with their own MGT/MDT on two MDS servers and their OSTs mounted on the same OSS?
3:53 a.m.
Hi,
does the configuration below make any sense, is it allowed by lustre?
The setup with lustre-2.5.3 (intel):
Two separate Lustre filesystems, fs1 and fs2, each having it's own set of
servers for separate MGT/MDT, mds01 and mds02, respectively
Question:
Can one have OSTs created on SAN block devices belonging to these two
Lustre filesystems mounted on the same OSS server?
Configuration details:
mds01 # mkfs.lustre --reformat --mgs /dev/sdb
mds01 # mkfs.lustre --reformat --fsname=fs1 --mdt --index=0
--mgsnode=mds01@tcp0 /dev/sdc
mds02 # mkfs.lustre --reformat --mgs /dev/sdb
mds02 # mkfs.lustre --reformat --fsname=fs2 --mdt --index=0
--mgsnode=mds02@tcp0 /dev/sdc
The OST SAN volumes are presented to both oss01 and oss02 servers:
oss01 # mkfs.lustre --reformat --fsname=fs1 --ost --index=0
--failnode=oss02@tcp0 --mgsnode=mds01@tcp0 /dev/sdb
oss02 # mkfs.lustre --reformat --fsname=fs2 --ost --index=0
--failnode=oss01@tcp0 --mgsnode=mds02@tcp0 /dev/sdc
oss02 # mkfs.lustre --reformat --fsname=fs2 --ost --index=1
--failnode=oss01@tcp0 --mgsnode=mds02@tcp0 /dev/sdd
I will mount the filesystems like this:
mds01 # mkdir -p /lustre/fs1-mgt01; mount -t lustre /dev/sdb
/lustre/fs1-mgt01
mds01 # mkdir -p /lustre/fs1-mdt01; mount -t lustre /dev/sdc
/lustre/fs1-mdt01
mds02 # mkdir -p /lustre/fs2-mgt01; mount -t lustre /dev/sdb
/lustre/fs2-mgt01
mds02 # mkdir -p /lustre/fs2-mdt01; mount -t lustre /dev/sdc
/lustre/fs2-mdt01
oss01 # mkdir -p /lustre/fs1-ost01; mount -t lustre /dev/sdb
/lustre/fs1-ost01
oss02 # mkdir -p /lustre/fs2-ost01; mount -t lustre /dev/sdc
/lustre/fs2-ost01
oss02 # mkdir -p /lustre/fs2-ost02; mount -t lustre /dev/sdd
/lustre/fs2-ost02
oss02 # umount /lustre/fs2-ost02
oss01 # mkdir -p /lustre/fs2-ost02; mount -t lustre /dev/sdd
/lustre/fs2-ost02
As the result oss01 will have fs1-ost01 and fs2-ost02, and oss02 fs2-ost01.
Vagrantfile for this setup is available at
https://gist.github.com/marcindulak/51fa898843b82373cf91
Cheers,
Marcin
2:33 p.m.
The OSTs themselves on the shared OSS don't care if they as connected to the same
filesystem or different ones. The clients don't care either. There _may_ be a bit of
difficulty with the OSS connecting to two different MGS nodes from the same server, since
we've never tried that before (AFAIK).
The more common config would have a single MGS and MDS with two different MDTs, one for
each filesystem, and separate OSTs for each filesystem on the same OSS.
So, in summary, what you describe should work, but you won't know until you try it.
Cheers, Andreas
On Dec 20, 2015, at 04:53, Marcin Dulak
<marcin.dulak@gmail.com<mailto:marcin.dulak@gmail.com>> wrote:
Hi,
does the configuration below make any sense, is it allowed by lustre?
The setup with lustre-2.5.3 (intel):
Two separate Lustre filesystems, fs1 and fs2, each having it's own set of servers for
separate MGT/MDT, mds01 and mds02, respectively
Question:
Can one have OSTs created on SAN block devices belonging to these two Lustre filesystems
mounted on the same OSS server?
Configuration details:
mds01 # mkfs.lustre --reformat --mgs /dev/sdb
mds01 # mkfs.lustre --reformat --fsname=fs1 --mdt --index=0 --mgsnode=mds01@tcp0 /dev/sdc
mds02 # mkfs.lustre --reformat --mgs /dev/sdb
mds02 # mkfs.lustre --reformat --fsname=fs2 --mdt --index=0 --mgsnode=mds02@tcp0 /dev/sdc
The OST SAN volumes are presented to both oss01 and oss02 servers:
oss01 # mkfs.lustre --reformat --fsname=fs1 --ost --index=0 --failnode=oss02@tcp0
--mgsnode=mds01@tcp0 /dev/sdb
oss02 # mkfs.lustre --reformat --fsname=fs2 --ost --index=0 --failnode=oss01@tcp0
--mgsnode=mds02@tcp0 /dev/sdc
oss02 # mkfs.lustre --reformat --fsname=fs2 --ost --index=1 --failnode=oss01@tcp0
--mgsnode=mds02@tcp0 /dev/sdd
I will mount the filesystems like this:
mds01 # mkdir -p /lustre/fs1-mgt01; mount -t lustre /dev/sdb /lustre/fs1-mgt01
mds01 # mkdir -p /lustre/fs1-mdt01; mount -t lustre /dev/sdc /lustre/fs1-mdt01
mds02 # mkdir -p /lustre/fs2-mgt01; mount -t lustre /dev/sdb /lustre/fs2-mgt01
mds02 # mkdir -p /lustre/fs2-mdt01; mount -t lustre /dev/sdc /lustre/fs2-mdt01
oss01 # mkdir -p /lustre/fs1-ost01; mount -t lustre /dev/sdb /lustre/fs1-ost01
oss02 # mkdir -p /lustre/fs2-ost01; mount -t lustre /dev/sdc /lustre/fs2-ost01
oss02 # mkdir -p /lustre/fs2-ost02; mount -t lustre /dev/sdd /lustre/fs2-ost02
oss02 # umount /lustre/fs2-ost02
oss01 # mkdir -p /lustre/fs2-ost02; mount -t lustre /dev/sdd /lustre/fs2-ost02
As the result oss01 will have fs1-ost01 and fs2-ost02, and oss02 fs2-ost01.
Vagrantfile for this setup is available at
https://gist.github.com/marcindulak/51fa898843b82373cf91
Cheers,
Marcin
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:19 p.m.
On Sun, Dec 20, 2015 at 11:33 PM, Dilger, Andreas <andreas.dilger(a)intel.com>
wrote:
The OSTs themselves on the shared OSS don't care if they as
connected to
the same filesystem or different ones. The clients don't care either. There
_may_ be a bit of difficulty with the OSS connecting to two different MGS
nodes from the same server, since we've never tried that before (AFAIK).
this setup works for me using a virtual test environment, but it seems
that I need
to mount
all OSTs belonging to fs1 on oss01, and all OSTs belonging to fs2 on oss02
the very first time.
After the initial mount it seems I can move OSTs between OSS freely.
If I try to mount e.g. /lustre/fs2-ost02 on oss01 before mounting it on
oss02 first, I get:
mount /dev/sdd at /lustre/fs2-ost02 failed: Cannot assign requested address
The more common config would have a single MGS and MDS with two
different
MDTs, one for each filesystem, and separate OSTs for each filesystem on the
same OSS.
in this setup all targets for each filesystem would be
on a separate SAN: MGT/MGT per filesystem and all OSTs.
In this way if one SAN goes down I don't miss all the Lustre filesystems,
what would happen if I had
only one MGT. I want to share OSTs between OSS because I don't have enough
OSS servers,
and moreover the load on different Lustre filesystems is different so I
hope to make savings
on the number of OSS servers that way.
So, in summary, what you describe should work, but you won't know until
you try it.
I'm just hesitating if I should put this into production, but if this
happens I let you know.
Is there anything special related to this configuration I should try to
test?
Cheers,
Marcin
Cheers, Andreas
On Dec 20, 2015, at 04:53, Marcin Dulak <marcin.dulak(a)gmail.com> wrote:
Hi,
does the configuration below make any sense, is it allowed by lustre?
The setup with lustre-2.5.3 (intel):
Two separate Lustre filesystems, fs1 and fs2, each having it's own set of
servers for separate MGT/MDT, mds01 and mds02, respectively
Question:
Can one have OSTs created on SAN block devices belonging to these two
Lustre filesystems mounted on the same OSS server?
Configuration details:
mds01 # mkfs.lustre --reformat --mgs /dev/sdb
mds01 # mkfs.lustre --reformat --fsname=fs1 --mdt --index=0
--mgsnode=mds01@tcp0 /dev/sdc
mds02 # mkfs.lustre --reformat --mgs /dev/sdb
mds02 # mkfs.lustre --reformat --fsname=fs2 --mdt --index=0
--mgsnode=mds02@tcp0 /dev/sdc
The OST SAN volumes are presented to both oss01 and oss02 servers:
oss01 # mkfs.lustre --reformat --fsname=fs1 --ost --index=0
--failnode=oss02@tcp0 --mgsnode=mds01@tcp0 /dev/sdb
oss02 # mkfs.lustre --reformat --fsname=fs2 --ost --index=0
--failnode=oss01@tcp0 --mgsnode=mds02@tcp0 /dev/sdc
oss02 # mkfs.lustre --reformat --fsname=fs2 --ost --index=1
--failnode=oss01@tcp0 --mgsnode=mds02@tcp0 /dev/sdd
I will mount the filesystems like this:
mds01 # mkdir -p /lustre/fs1-mgt01; mount -t lustre /dev/sdb
/lustre/fs1-mgt01
mds01 # mkdir -p /lustre/fs1-mdt01; mount -t lustre /dev/sdc
/lustre/fs1-mdt01
mds02 # mkdir -p /lustre/fs2-mgt01; mount -t lustre /dev/sdb
/lustre/fs2-mgt01
mds02 # mkdir -p /lustre/fs2-mdt01; mount -t lustre /dev/sdc
/lustre/fs2-mdt01
oss01 # mkdir -p /lustre/fs1-ost01; mount -t lustre /dev/sdb
/lustre/fs1-ost01
oss02 # mkdir -p /lustre/fs2-ost01; mount -t lustre /dev/sdc
/lustre/fs2-ost01
oss02 # mkdir -p /lustre/fs2-ost02; mount -t lustre /dev/sdd
/lustre/fs2-ost02
oss02 # umount /lustre/fs2-ost02
oss01 # mkdir -p /lustre/fs2-ost02; mount -t lustre /dev/sdd
/lustre/fs2-ost02
As the result oss01 will have fs1-ost01 and fs2-ost02, and oss02
fs2-ost01.
Vagrantfile for this setup is available at
https://gist.github.com/marcindulak/51fa898843b82373cf91
Cheers,
Marcin
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
1938
days inactive
1939
days old
2 comments
2 participants
participants (2)
-
Dilger, Andreas -
Marcin Dulak