syzbot


possible deadlock in ocfs2_finish_quota_recovery

Status: fixed on 2025/06/06 09:23
Bug presence: origin:upstream
[Documentation on labels]
Reported-by: syzbot+[email protected]
Fix commit: 4c3a0b0b23dd ocfs2: stop quota recovery before disabling quotas
First crash: 139d, last: 45d
Fix bisection: fixed by (bisect log) :
commit 4c3a0b0b23dd9639739732556830a1d2fe14dc60
Author: Jan Kara <[email protected]>
Date: Thu Apr 24 13:45:13 2025 +0000

  ocfs2: stop quota recovery before disabling quotas

  
Bug presence (1)
Date Name Commit Repro Result
2025/03/16 upstream (ToT) d1275e99d1c4 C [report] possible deadlock in deactivate_super
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 possible deadlock in ocfs2_finish_quota_recovery 11 63d 141d 0/3 upstream: reported on 2025/01/30 02:34
upstream possible deadlock in ocfs2_finish_quota_recovery ocfs2 C 186 44d 137d 0/29 upstream: reported C repro on 2025/02/02 09:01

Sample crash report:
ocfs2: Finishing quota recovery on device (7,1) for slot 0
======================================================
WARNING: possible circular locking dependency detected
6.1.137-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:3/46 is trying to acquire lock:
ffff88801e3020e0 (&type->s_umount_key#64){++++}-{3:3}, at: ocfs2_finish_quota_recovery+0x129/0x1e40 fs/ocfs2/quota_local.c:600

but task is already holding lock:
ffffc90000b77d00 ((work_completion)(&journal->j_recovery_work)){+.+.}-{0:0}, at: process_one_work+0x7a1/0x1160 kernel/workqueue.c:2267

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&journal->j_recovery_work)){+.+.}-{0:0}:
       process_one_work+0x7d9/0x1160 kernel/workqueue.c:2268
       worker_thread+0xaa2/0x1250 kernel/workqueue.c:2439
       kthread+0x29d/0x330 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

-> #1 ((null)){+.+.}-{0:0}:
       __flush_workqueue+0x13d/0x1380 kernel/workqueue.c:2812
       ocfs2_shutdown_local_alloc+0xe4/0xa10 fs/ocfs2/localalloc.c:379
       ocfs2_dismount_volume+0x1f4/0x920 fs/ocfs2/super.c:1879
       generic_shutdown_super+0x130/0x340 fs/super.c:501
       kill_block_super+0x7c/0xe0 fs/super.c:1470
       deactivate_locked_super+0x93/0xf0 fs/super.c:332
       cleanup_mnt+0x463/0x4f0 fs/namespace.c:1186
       task_work_run+0x1ca/0x250 kernel/task_work.c:203
       resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
       exit_to_user_mode_loop+0xe6/0x110 kernel/entry/common.c:177
       exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:210
       __syscall_exit_to_user_mode_work kernel/entry/common.c:292 [inline]
       syscall_exit_to_user_mode+0x16/0x40 kernel/entry/common.c:303
       do_syscall_64+0x58/0xa0 arch/x86/entry/common.c:87
       entry_SYSCALL_64_after_hwframe+0x68/0xd2

-> #0 (&type->s_umount_key#64){++++}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3090 [inline]
       check_prevs_add kernel/locking/lockdep.c:3209 [inline]
       validate_chain kernel/locking/lockdep.c:3825 [inline]
       __lock_acquire+0x2cf8/0x7c50 kernel/locking/lockdep.c:5049
       lock_acquire+0x1b4/0x490 kernel/locking/lockdep.c:5662
       down_read+0x42/0x2d0 kernel/locking/rwsem.c:1520
       ocfs2_finish_quota_recovery+0x129/0x1e40 fs/ocfs2/quota_local.c:600
       ocfs2_complete_recovery+0x157c/0x23a0 fs/ocfs2/journal.c:1324
       process_one_work+0x898/0x1160 kernel/workqueue.c:2292
       worker_thread+0xaa2/0x1250 kernel/workqueue.c:2439
       kthread+0x29d/0x330 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

other info that might help us debug this:

Chain exists of:
  &type->s_umount_key#64 --> (null) --> (work_completion)(&journal->j_recovery_work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&journal->j_recovery_work));
                               lock((null));
                               lock((work_completion)(&journal->j_recovery_work));
  lock(&type->s_umount_key#64);

 *** DEADLOCK ***

2 locks held by kworker/u4:3/46:
 #0: ffff888073c43938 ((wq_completion)ocfs2_wq#2){+.+.}-{0:0}, at: process_one_work+0x7a1/0x1160 kernel/workqueue.c:2267
 #1: ffffc90000b77d00 ((work_completion)(&journal->j_recovery_work)){+.+.}-{0:0}, at: process_one_work+0x7a1/0x1160 kernel/workqueue.c:2267

stack backtrace:
CPU: 1 PID: 46 Comm: kworker/u4:3 Not tainted 6.1.137-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/19/2025
Workqueue: ocfs2_wq ocfs2_complete_recovery
Call Trace:
 <TASK>
 dump_stack_lvl+0x168/0x22e lib/dump_stack.c:106
 check_noncircular+0x274/0x310 kernel/locking/lockdep.c:2170
 check_prev_add kernel/locking/lockdep.c:3090 [inline]
 check_prevs_add kernel/locking/lockdep.c:3209 [inline]
 validate_chain kernel/locking/lockdep.c:3825 [inline]
 __lock_acquire+0x2cf8/0x7c50 kernel/locking/lockdep.c:5049
 lock_acquire+0x1b4/0x490 kernel/locking/lockdep.c:5662
 down_read+0x42/0x2d0 kernel/locking/rwsem.c:1520
 ocfs2_finish_quota_recovery+0x129/0x1e40 fs/ocfs2/quota_local.c:600
 ocfs2_complete_recovery+0x157c/0x23a0 fs/ocfs2/journal.c:1324
 process_one_work+0x898/0x1160 kernel/workqueue.c:2292
 worker_thread+0xaa2/0x1250 kernel/workqueue.c:2439
 kthread+0x29d/0x330 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>

Crashes (12):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/05/05 12:38 linux-6.1.y ac7079a42ea5 6ca47dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/05/05 12:38 linux-6.1.y ac7079a42ea5 6ca47dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/04/07 09:27 linux-6.1.y 3dfebb87d7eb 1c65791e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/03/27 14:39 linux-6.1.y 344a09659766 6c09fb82 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/03/23 23:13 linux-6.1.y 344a09659766 875573af .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/03/20 19:10 linux-6.1.y 344a09659766 62330552 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/02/20 19:11 linux-6.1.y 0cbb5f65e52f 0808a665 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/02/15 06:09 linux-6.1.y 0cbb5f65e52f 40a34ec9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/02/14 02:00 linux-6.1.y 0cbb5f65e52f d9a046cf .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/02/06 02:45 linux-6.1.y 0cbb5f65e52f 577d049b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/02/01 06:15 linux-6.1.y 75cefdf153f5 aa47157c .config console log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/01/31 16:26 linux-6.1.y 75cefdf153f5 aa47157c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in ocfs2_finish_quota_recovery
* Struck through repros no longer work on HEAD.
OSZAR »