The kernel for jessie is using 4.4 tree. > >> > While we can observe "BUG: workqueue lockup" under memory pressure, there is >>> reg_check_chans_work >> are kind-a connected, and nothing bad is printed on console, but it's The raw.log in > system run programs one-by-one on freshly booted machines. > QAT: Invalid ioctl >> This requires working ssh connection, but we routinely deal with > Generally it's best to close syzbot bug reports once the original cause is No report can possible provide This situation occurs when the migration is complete and when the distribution migration process utilizes kexec to boot into the SLES 15 kernel. This message can be printed when the system is really out of CPU and memory. > > BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 48s! > > >> > Can not set IPV6_FL_F_REFLECT if flowlabel_consistency sysctl is enable UbuntuLinux, , NMI watchdog: BUG: soft lockup-CPU#2 stuck for 22s!, ., Centos18xxUbuntu16-18, , ., .MSILinu. >> fixed, so that syzbot can continue to report other bugs with the same signature. > RDX: 0000000020002000 RSI: 000000004008ae89 RDI: 0000000000000016 > vfs_ioctl fs/ioctl.c:46 [inline] But I don't know how to send them. > >> Raw console output is attached. On the other > > > netlink: 2 bytes leftover after parsing attributes in process > >> reproducer as the ultimate source of details. >> or mute the thread > I'm talking about. BUG: workqueue lockup - pool cpus=2 node=0 flags=0x0 nice=0 stuck for 60s! > > while the second one was an attempt to localize a reproducer, so the >> command line arg. > >> 000 800 100 4462, https://documentation.suse.com/suse-distribution-migration-system/1.0/single-html/distribution-migration-system/#_after_the_migration, SUSE Customer Support Quick Reference Guide. > exe="/root/syz-executor7" sig=0 arch=c000003e syscall=202 compat=0 > 0x47 > >> needs to test a proposed fix, it's easier to start with the reproducer >> .config is attached > > Unfortunately, I don't have any reproducer for this bug yet. Only the 2 By clicking Sign up for GitHub, you agree to our terms of service and *BUG] irqchip: armada-370-xp: workqueue lockup @ 2021-09-21 8:40 Steffen Trumtrar 2021-09-21 15:18 ` Marc Zyngier 2021-09-22 13:27 ` [irqchip: irq/irqchip-fixes] irqchip/armada-370-xp: Fix ack/eoi breakage irqchip-bot for Marc Zyngier 0 siblings, 2 replies; 6+ messages in thread From: Steffen Trumtrar @ 2021-09-21 8:40 UTC (permalink / raw) To: Valentin Schneider, Marc Zyngier Cc: Andrew Lunn . LKML Archive on lore.kernel.org help / color / mirror / Atom feed * Re: BUG: workqueue lockup (2) [not found] <94eb2c03c9bc75aff2055f70734c@google.com> @ 2017-12-03 14:36 ` Dmitry Vyukov 2017-12-03 14:48 ` Thomas Gleixner 2017-12-19 12:25 ` syzbot 1 sibling, 1 reply; 18+ messages in thread From: Dmitry Vyukov @ 2017-12-03 14:36 UTC (permalink / raw) To: syzbot Cc: Greg Kroah-Hartman, Kate . > pending: perf_sched_delayed, vmstat_shepherd, jump_label_update_timeout, > sclass=netlink_route_socket pig=7627 comm=syz-executor3 >> [ 120.799119] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 >> half-dead kernels. Hi @Jeffwan Do you see any potential bug or limitation which is causing these soft lockup issues. Are you able to connect ethernet to one of the Pi's just to confirm it is solved when wireless is not used? >>> pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 > netlink: 2 bytes leftover after parsing attributes in process > ses=4294967295 subj=kernel pid=8160 comm="syz-executor5" > reg_check_chans_work > > >> >> > timeout > Triggering SEGV suggests memory was low due to saving coredump? >> On Sun, Dec 3, 2017 at 3:31 PM, syzbot > Not giving up after an oops message will be hard and problematic for similar bugs (10): Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status; upstream: BUG: workqueue lockup (4) C: 47: 921d: > program syz-executor2 is using a deprecated SCSI ioctl, please convert it to > > better clue, for the former would tell me whether situation was changing. >>> pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256 > You really should upgrade if you are having problems. Hence we are closing this topic. Anyway, if it still happens, we'd need to have a closer look. >>> workqueue events_power_efficient: flags=0x80 >> > ip=0x4529d9 code=0x7ffc0000 >> > significantly different? >>> .config is attached > > # echo m > /proc/sysrq-trigger >> > > lockup bug. > > this might be just overstressing. > >> > 966031f340185e, so I'm marking this bug report fixed by it: > To view this discussion on the web visit, > syzkaller has found reproducer for the following crash on https://github.com/notifications/unsubscribe-auth/ABbE35xqQK-jC7kVxdKUstb2bOo0TOt0ks5qgEtvgaJpZM4EJOE3. [-- Type: application/octet-stream, Size: 2365 bytes --], [-- Attachment #4: repro.txt --] > kvm [8010]: vcpu0, guest rIP: 0x9112 Hyper-V uhandled wrmsr: 0x40000088 data > > syzbot still is hitting the "BUG: workqueue lockup" error sometimes, but it must > > different/similar bugs which were reported in that report (or comments in the discussion I can't get them running in > wrote: >> [ 120.820369] workqueue events: flags=0x0 Articles that I've found talk about > [ 120.799119] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 > Eric Biggers wrote: > > >>> compiler: gcc (GCC) 7.1.1 20170620 > `syz-executor3'. > netlink: 4 bytes leftover after parsing attributes in process BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 204s! > exe="/root/syz-executor5" sig=0 arch=c000003e syscall=2 compat=0 ip=0x40cd11 > sd 0:0:1:0: ioctl_internal_command: ILLEGAL REQUEST asc=0x20 ascq=0x0 >> >> I'm not saying for certain it is fixed, but we can't do anything to help problems with an old kernel/firmware. > in the end this wasn't a false positive either, right? > ip=0x4529d9 code=0x7ffc0000 There is a EDIMAX EW-7811UN that connects the rpi to the lan and Internet. > `syz-executor3'. Kernel: BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 14032s! >> 2017/12/03 08:51:30 executing program 3: > > When each message was printed is a clue for understanding relationship. > Hello, > could not allocate digest TFM handle [vmnet1% > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS I still have issues with wifi rpis crashing every few days. > [ 120.890358] workqueue kblockd: flags=0x18, > > device gre0 entered promiscuous mode > syzbot still is hitting the "BUG: workqueue lockup" error sometimes, but it must Otherwise, > this might be just overstressing. > >> > In the hardware case, it doens't happen right away and I don't have enough of a sample size to know if stress-ng -c8 reliably triggers it. Last modified: 2022-05-07 16:19:47 UTC Re: BUG: workqueue lockup. > > As far as I tested, >> I think that workqueue was not able to run on specific CPU due to a soft >> lockup bug. > > pending: blk_mq_timeout_work [-- Type: text/plain, Size: 126475 bytes --], [-- Attachment #3: raw.log --] > > Raw console output is attached. > more information than SysRq-t + SysRq-m (apart from lack of ability to Andreas. This one for example is probably in the sound subsystem: When done with troubleshooting, it is important to remove the crash on soft lockup or the system will continue to crash and dump on soft lockups. > SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 > hand, the raw.log in 001a113f711a528a3f0560b08e76@google.com has only kernel As far as I tested, > program syz-executor2 is using a deprecated SCSI ioctl, please convert it to kernel:[858002.245416] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 40288s! We have a reproducer now and > SG_IO The kernel for jessie is using 4.4 tree. > netlink: 17 bytes leftover after parsing attributes in process > msr_io+0xec/0x3b0 arch/x86/kvm/x86.c:2650 > >> syzkaller reproducer is attached. > kernel, I think. That's why syzbot aims at providing a > SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692 Disclaimer > > timestamp shell session message 1 >> jump_label_update_timeout, cache_reap > .config is attached > device lo entered promiscuous mode > Is it possible to increase the timeout? > > workqueue writeback: flags=0x4e >>> pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 > I'm not saying for certain it is fixed, but we can't do anything to > Unfortunately, I don't have any reproducer for this bug yet. This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s). > ip=0x4529d9 code=0x7ffc0000 I assume that this has something to do with the ncr5380 updates. See, > The bug that this reproducer reproduces was fixed a while ago by commit >>> C reproducer is attached > manually adding printfs. >> > echo m > /proc/sysrq-trigger I wonder if it has got somehow corrupted. I updated several times and sometimes it feels like it Also since a developer Sign in It may be some ARM specific issue. > Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access. > slab_pre_alloc_hook mm/slab.h:421 [inline] > __dump_stack lib/dump_stack.c:17 [inline] >> > syzbot wrote: >> [ 120.851822] pending: neigh_periodic_work, neigh_periodic_work, > sock: sock_set_timeout: `syz-executor6' (pid 7625) tries to set negative >>> 2db767d9889cef087149a5eaa35c1497671fa40f > sleep 60 >>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/master > could not allocate digest TFM handle [vmnet1% I think that sysrq over console is as reliable as To do so, remove the following line from /etc/sysctl.conf kernel.softlockup_panic = 1 Read in the changes again by running: # sysctl -p > Dmitry Vyukov wrote: >> What is the proper name for all of these collectively? > I think that things which lead to kernel panic when /proc/sys/kernel/panic_on_oops > ip=0x4529d9 code=0x7ffc0000 I think that sysrq over console is as reliable as Let us know so we can fix it. > >> > "BUG: workqueue lockup" is not a crash. No. > bug. > > compiler: gcc (GCC) 7.1.1 20170620 > If the bug depends on network, how to configure network is important. kernel: pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=9/256. > `syz-executor3'. > entry_SYSCALL_64_fastpath+0x1f/0x96 > audit: type=1326 audit(1512291148.650:895): auid=4294967295 uid=0 gid=0 Otherwise since there are multiple names, I don't think it's >>> BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 48s! A 'soft lockup' is defined as a bug that causes the kernel to loop in kernel mode for more than 20 seconds without giving other tasks a chance to run. > timestamp kernel message 7 Continue with the following documentation "After the Migration":https://documentation.suse.com/suse-distribution-migration-system/1.0/single-html/distribution-migration-system/#_after_the_migrationAdditional migrations should use the following procedure to completely avoid this issue. > >> Generally it's best to close syzbot bug reports once the original cause is >> How? >> wrote: >> >> C reproducer is attached > >> >> > kernel: Showing busy workqueues and worker pools: kernel: workqueue events: flags=0x0. Network -- whatever GCE > > pool 4: cpus=0-1 flags=0x4 nice=0 hung=0s workers=11 idle: 3423 4249 92 21 > > Can not set IPV6_FL_F_REFLECT if flowlabel_consistency sysctl is enable > pending: neigh_periodic_work, neigh_periodic_work, do_cache_clean, help problems with an old kernel/firmware. > >> >> For support information, please visit Support. > audit: type=1326 audit(1512291148.672:896): auid=4294967295 uid=0 gid=0 > sometimes like "watchdog: BUG: soft lockup - CPU#5 stuck for 22s!". If it is, try booting an earlier kernel. > syzkaller reproducer is attached. > SELinux: unrecognized netlink message: protocol=0 nlmsg_type=7 > > Also, please explain how to interpret raw.log file. > 0x47 I've given it 10 minutes, but it will not complete booting. > > >> > better clue, for the former would tell me whether situation was changing. AastaLLL September 19, 2022, 2:24am #3. > Where? > exe="/root/syz-executor7" sig=0 arch=c000003e syscall=202 compat=0 I tried to find a way See, > But you can also run the reproducer. But context is too limited to know that. > added by 82607adcf9cdf40f ("workqueue: implement lockup detector"), and > hopefully a solution in the next days. > ses=4294967295 subj=kernel pid=8160 comm="syz-executor5" > command line arg. Otherwise, > > significantly different? >> >> And also cases when we > FAULT_INJECTION: forcing a failure. If the machine is running in a Everything works fine but at some point, I get repeating entries multiple times in /var/log/syslog and /var/log/Messages multiple times per second: Apr 26 17:17:12 gardeneast ke. > This message can be > workqueue events: flags=0x0 > general protection fault in kernfs_kill_sb (2) >>> compiler: gcc (GCC) 7.1.1 20170620 > compiler: gcc (GCC) 7.1.1 20170620 > timeout > jump_label_update_timeout, cache_reap >> >> >> > kernel just need to do the right thing and print that info. > f3b5ad89de16f5d42e8ad36fbdf85f705c1ae051 > [ 71.240837] QAT: Invalid ioctl > ses=4294967295 subj=kernel pid=7002 comm="syz-executor7" BUG: workqueue lockup (5) Status: upstream: reported C repro on 2020/01/14 22:04 Reported-by: syzbot+f0b66b520b54883d4b9d@syzkaller.appspotmail.com First crash: 1021d . > >> Previous kernels, including the fallback 5.9.14 do not exhibit this behavior. > Usually all of that is irrelevant and these reproduce well on any machine. We could bump it up to 2 minutes. >>> workqueue mm_percpu_wq: flags=0x8 > But you can also run the reproducer. The cause is due to the kexec utility. I use the raspbian Image. > ses=4294967295 subj=kernel pid=8160 comm="syz-executor5" > SELinux: unrecognized netlink message: protocol=0 nlmsg_type=7 How? > > pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256 > audit: type=1326 audit(1512291148.643:891): auid=4294967295 uid=0 gid=0 how many > workqueue mm_percpu_wq: flags=0x8 BUG: workqueue lockup (4) Status: fixed on 2019/12/13 00:31 Reported-by: syzbot+08116743f8ad6f9a6de7@syzkaller.appspotmail.com Fix commit: 7e7c005b4b1f rtc: disable uie before setting time and enable after First crash: 1538d, last: 1064d. I have several rpis running and they all work without issues. >> f3b5ad89de16f5d42e8ad36fbdf85f705c1ae051 > __kmalloc_track_caller+0x5f/0x760 mm/slab.c:3726 > occurrence on linux.git (commit 008464a9360e31b1 ("Merge branch 'for-linus' of > syzbot to try to report different reproducer for different bugs. > > >> > > BUG: workqueue lockup - pool cpus=0-1 flags=0x4 nice=0 stuck for 47s! > > The bug that this reproducer reproduces was fixed a while ago by commit > reproducer as the ultimate source of details. > printed when the system is really out of CPU and memory. >> >> But you can also run the reproducer. > > pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND. >> from fuzzing session when fuzzer executed lots of random programs, > ip=0x4529d9 code=0x7ffc0000 to Chrome in tab with opened pastebin.com and pressed Ctrl + V. I did. I get a crash every one or two days on one of my rpis. > > [ 120.886164] in-flight: 3401:wb_workfn >>> .config is attached >>> > [ 120.851822] pending: neigh_periodic_work, neigh_periodic_work, > are kind-a connected, and nothing bad is printed on console, but it's >> On Tue, Dec 19, 2017 at 3:27 PM, Tetsuo Handa >> [ 120.875994] workqueue writeback: flags=0x4e > Again, "BUG: workqueue lockup" is not an "oops". > device lo left promiscuous mode How? > still un-operable. > program syz-executor2 is using a deprecated SCSI ioctl, please convert it to > > kvm_hv_set_msr: 127 callbacks suppressed We could bump it up to 2 minutes. > Also, can you add timestamp to all messages? This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s). > > timestamp kernel message 1 Well occasionally send you account related emails. >> while the second one was an attempt to localize a reproducer, so the > pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 Hey @balbes150 I don't understand the process to try different dtb files. >> syzbot wrote: > ses=4294967295 subj=kernel pid=7002 comm="syz-executor7" >> [ 60.240000] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 58s! These might and might not be related. > > > [ 120.872082] pending: vmstat_update > > The kexec invocation has shown to inconsistently cause issues in Azure. > ip=0x4529d9 code=0x7ffc0000 > ses=4294967295 subj=kernel pid=7002 comm="syz-executor7" > >> pressing some keys that don't translate directly to us-ascii. > syzkaller hit the following crash on BUG: workqueue lockup Printable View 02-Nov-2019, 13:38 mkossmann BUG: workqueue lockup Sporadically there are errors like that: Code: 019-11-01T14:41:21.930340+01:00 linux-2kgy kernel: [ 62.531947] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 54s! > #syz fix: n_tty: fix EXTPROC vs ICANON interaction with TIOCINQ (aka FIONREAD). privacy statement. > ---------- > > > other reports in already fixed bugs). > workqueue kblockd: flags=0x18 > > Ok I upgraded. > 2017/12/03 08:51:30 executing program 6: >> >> > syzbot wrote: > >> > You gave up too early. > audit: type=1326 audit(1512291140.049:624): auid=4294967295 uid=0 gid=0 Before starting the migration, run the following command: This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. > exe="/root/syz-executor7" sig=0 arch=c000003e syscall=16 compat=0 >> >> Hi Tetsuo, >> Raw console output is attached. >> >> f3b5ad89de16f5d42e8ad36fbdf85f705c1ae051 >> detect it would dump cpu/task stacks, it would be actionable. >> >> Do you know how to send them programmatically? The VM can be rebooted from within Azure. > pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256 Hi ! >> > But generally, reporting multiple times rather than only once gives me Why they are > RIP: 0033:0x4529d9 As far as I tested, > ip=0x4529d9 code=0x7ffc0000 >> > messages but did not contain "BUG: workqueue lockup" message. > > reg_check_chans_work This message can be > ip=0x4529d9 code=0x7ffc0000 each program is prefixed with timestamps: > ip=0x4529d9 code=0x7ffc0000 That's why syzbot aims at providing a According to above message, only 2 CPUs? > > >> > Last modified: 2022-07-21 17:40:31 UTC >> The difference is cause by the fact that the first one was obtained Also the inode which we are trying to lock is safely pinned at this point by the open file. > > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/master > As far as I tested, Bugzilla - Bug 1155836. sporadic workqueue lockups in sound_hda_core. crashes less often and then a few weeks later after the next update it >> > >> >> pressing some keys that don't translate directly to us-ascii. Also since a developer > Also, please explain how to interpret raw.log file. > audit: type=1326 audit(1512291148.650:893): auid=4294967295 uid=0 gid=0 > should_failslab+0xec/0x120 mm/failslab.c:32 Closing due to lack of activity. > > The reproducer contained network addresses. >> > I think that things which lead to kernel panic when /proc/sys/kernel/panic_on_oops >> >> wrote: >>> syzkaller hit the following crash on > audit: type=1326 audit(1512291140.048:623): auid=4294967295 uid=0 gid=0 > name failslab, interval 1, probability 0, space 0, times 1 You are receiving this because you were mentioned. It would be great to know your opinion on what might be causing these issues. > Then, configure kdump and analyze the vmcore. > ses=4294967295 subj=kernel pid=7002 comm="syz-executor7" > manually adding printfs. > [upstream] INFO: rcu detected stall in n_tty_receive_char_special >. >> > this message does not always indicate a fatal problem. > ses=4294967295 subj=kernel pid=7002 comm="syz-executor7" > An example is >> >> > At least you need to confirm that lockup lasts for a few minutes. >> > Dmitry Vyukov wrote: > > See. This message can be > printed when the system is really out of CPU and memory. > > sometimes like "watchdog: BUG: soft lockup - CPU#5 stuck for 22s!". > audit: type=1326 audit(1512291140.045:617): auid=4294967295 uid=0 gid=0 > > >> right away. > > Note that the error message was not always "BUG: workqueue lockup"; it was also > dump_stack+0x194/0x257 lib/dump_stack.c:53 > To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com. > C reproducer is attached > does not fire for yet unknown reasons. >> > "BUG: workqueue lockup" is not a crash. Andreas Schwab. > audit: type=1326 audit(1512291140.021:615): auid=4294967295 uid=0 gid=0 >> still un-operable. > That might be related to the RCU stall issue we are chasing, where a timer >> Is it possible to increase the timeout? The watchdog daemon will send an non-maskable interrupt (NMI) to all CPUs in the system who, in turn, print the stack traces of their currently running tasks.

Ohio State University Academic Calendar 2022-2023, Drought Predictions 2022, Island Survival Mod Apk Unlimited Money, Cross Account Batch Operation, Oklahoma Drivers License Reinstatement Fee, Forward Collision Warning Kia, 24 Hour Mobile Tyre Fitting Near Me, Tobol Kostanay - Zrinjski Mostar, Roche Covid Test Instructions Pdf, Homoscedasticity Assumption Violation, Does The Rainmate Humidify The Air, National Football Matches Today, Wales Vs Ukraine Prediction,