11:55 a.m.
Hi!
If I try to change the mode of a devdax namespace that's in use (mapped by
some process), ndctl hangs:
[ 9546.754673] INFO: task ndctl:3907 blocked for more than 1208 seconds.
[ 9546.754677] Not tainted 5.3.0-00048-g7f09b8bce091 #1
[ 9546.754679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[ 9546.754681] ndctl D 0 3907 3856 0x00004004
[ 9546.754684] Call Trace:
[ 9546.754689] ? __schedule+0x281/0x670
[ 9546.754692] ? __switch_to_asm+0x34/0x70
[ 9546.754694] ? __switch_to_asm+0x34/0x70
[ 9546.754696] schedule+0x39/0xa0
[ 9546.754699] schedule_timeout+0x22b/0x320
[ 9546.754701] ? __switch_to_asm+0x34/0x70
[ 9546.754703] ? __switch_to_asm+0x40/0x70
[ 9546.754705] ? __switch_to_asm+0x34/0x70
[ 9546.754707] ? __switch_to+0x162/0x440
[ 9546.754710] ? apic_timer_interrupt+0xa/0x20
[ 9546.754712] wait_for_completion+0x100/0x150
[ 9546.754714] ? wake_up_q+0x60/0x60
[ 9546.754718] dev_pagemap_cleanup+0x47/0x60
[ 9546.754720] devm_memremap_pages_release+0xc5/0x220
[ 9546.754724] release_nodes+0x221/0x270
[ 9546.754728] dax_pmem_compat_release+0x30/0x50 [dax_pmem_compat]
[ 9546.754730] ? dax_pmem_compat_remove+0x20/0x20 [dax_pmem_compat]
[ 9546.754733] device_for_each_child+0x57/0x90
[ 9546.754736] dax_pmem_compat_remove+0x13/0x20 [dax_pmem_compat]
[ 9546.754739] nvdimm_bus_remove+0x4e/0xc0
[ 9546.754741] device_release_driver_internal+0xd8/0x1b0
[ 9546.754743] unbind_store+0xff/0x130
[ 9546.754746] kernfs_fop_write+0x140/0x1b0
[ 9546.754749] vfs_write+0xe4/0x1d0
[ 9546.754751] ksys_write+0x70/0x100
[ 9546.754754] do_syscall_64+0x50/0x100
[ 9546.754756] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9546.754758] RIP: 0033:0x7f2d375dfad4
[ 9546.754762] Code: Bad RIP value.
[ 9546.754763] RSP: 002b:00007ffd61eca4e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 9546.754766] RAX: ffffffffffffffda RBX: 000055fb48f0982f RCX: 00007f2d375dfad4
[ 9546.754767] RDX: 0000000000000007 RSI: 000055fb48f0982f RDI: 0000000000000003
[ 9546.754769] RBP: 0000000000000007 R08: 00000000ffffffff R09: 00007ffd61eca3c0
[ 9546.754770] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003
[ 9546.754771] R13: 00007f2d37166e70 R14: 0000000000000000 R15: 000055fb48f0c900
Fake pmem (memmap=4G!16G), the command is:
ndctl create-namespace -e namespace0.0 -m fsdax -f
-f is needed as a label-less fake pmem namespace is always active.
According to the man page, reconfiguring in that case is not allowed (duh),
and the operation is supposed to gracefully fail.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
3:10 p.m.
On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte(a)angband.pl> wrote:
Hi!
If I try to change the mode of a devdax namespace that's in use (mapped by
some process), ndctl hangs:
Is it merely mapped, or might the pages be actively pinned / in use by
another part of the kernel? The kernel has no choice but to wait for
active page pins to drain. Can you get a stack trace of the process
with the dev-dax instance mapped?
3:47 p.m.
On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski
<kilobyte(a)angband.pl> wrote:
> Hi!
> If I try to change the mode of a devdax namespace that's in use (mapped by
> some process), ndctl hangs:
Is it merely mapped, or might the pages be actively pinned / in use by
another part of the kernel? The kernel has no choice but to wait for
active page pins to drain. Can you get a stack trace of the process
with the dev-dax instance mapped?
Looks like the behaviour is different depending on what the other process
is:
* with qemu, the hang is 100% reproducible, the guest continues to work and
cleanly exits -- qemu does not exit on its own (unlike normal case) but
SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is.
* with mere mmap() (I've used vmemcache) ndctl allows
reconfiguring the namespace. No hang.
My way to start qemu is:
.----
#!/bin/sh
NET="-net bridge -net nic"
DISK=eoan-devdax.disk
exec qemu-system-x86_64 -enable-kvm -m 4096,slots=2,maxmem=16G -smp 8 $NET \
-drive if=none,id=hd,file="$DISK",format=raw,cache=unsafe,discard=on \
-device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd \
-M pc,nvdimm,nvdimm-persistence=mem-ctrl \
-object
memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4225761280,align=2M,pmem=on
\
-device nvdimm,id=nvdimm1,memdev=mem1,label-size=256K \
-vnc :5
`----
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
3:50 p.m.
On Thu, Sep 19, 2019 at 8:47 AM Adam Borowski <kilobyte(a)angband.pl> wrote:
On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
> On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte(a)angband.pl> wrote:
> > Hi!
> > If I try to change the mode of a devdax namespace that's in use (mapped by
> > some process), ndctl hangs:
>
> Is it merely mapped, or might the pages be actively pinned / in use by
> another part of the kernel? The kernel has no choice but to wait for
> active page pins to drain. Can you get a stack trace of the process
> with the dev-dax instance mapped?
Looks like the behaviour is different depending on what the other process
is:
* with qemu, the hang is 100% reproducible, the guest continues to work and
cleanly exits -- qemu does not exit on its own (unlike normal case) but
SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is.
* with mere mmap() (I've used vmemcache) ndctl allows
reconfiguring the namespace. No hang.
My way to start qemu is:
.----
#!/bin/sh
NET="-net bridge -net nic"
DISK=eoan-devdax.disk
exec qemu-system-x86_64 -enable-kvm -m 4096,slots=2,maxmem=16G -smp 8 $NET \
-drive if=none,id=hd,file="$DISK",format=raw,cache=unsafe,discard=on \
-device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd \
-M pc,nvdimm,nvdimm-persistence=mem-ctrl \
-object
memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4225761280,align=2M,pmem=on
\
-device nvdimm,id=nvdimm1,memdev=mem1,label-size=256K \
-vnc :5
Ok, I'll take a look. At first glance nothing in that config should be
holding an indefinite page pin, so it does smell like a kernel bug.
3:50 p.m.
On Thu, Sep 19, 2019 at 05:47:08PM +0200, Adam Borowski wrote:
On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
> On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte(a)angband.pl> wrote:
> > If I try to change the mode of a devdax namespace that's in use (mapped by
> > some process), ndctl hangs:
>
> Is it merely mapped, or might the pages be actively pinned / in use by
> another part of the kernel? The kernel has no choice but to wait for
> active page pins to drain. Can you get a stack trace of the process
> with the dev-dax instance mapped?
Looks like the behaviour is different depending on what the other process
is:
* with qemu, the hang is 100% reproducible, the guest continues to work and
cleanly exits -- qemu does not exit on its own (unlike normal case) but
SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is.
Correction: not 100%. I just had qemu die with SIGBUS instead.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
1012
days inactive
1012
days old
4 comments
2 participants
participants (2)
-
Adam Borowski -
Dan Williams