(I apologize to those who are also lustre-discuss members as this is a
cross-post)
We have been using the following method to maintain a backup MGS/MDS:
https://jira.hpdd.intel.com/browse/LUDOC-161
The procedure we follow is commented by Scott and was taken from our own
internal documentation. Our method is to create a ZFS file systems:
lustre-meta/fsmeta
lustre-meta/mgs
And then we snapshot (-r), then zfs send (-R)/receive it. To use the
backup we swap its IP addresses and name with the primary.
If the backup is swapped and used immediately after the snapshot is
taken, this method works. If you continue to use the original server
before migrating to the backup, it does not work. This is the equivalent
of migrating to a not-perfectly-up-to-date snapshot.
In the broken case, we can still read files. We cannot create new files.
The MDS does show files we attempt to create but their attributes are
all unknown. We are unable to manipulate those files (rm, mv, etc)
because they do not exist on the OSTs. The error returned is "cannot
allocate memory" (see LU-4524).
We suspected the configuration logs and so we re-ran writeconf and
remounted. Same behavior.
We are running lustre 2.4.0 on the servers and have tested with 2.4.0
and 2.1.6 clients.
Best,
Jesse Stroik
University of Wisconsin